[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:22 dist-test-slave-wmd0 krb5kdc[14854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:22 dist-test-slave-wmd0 krb5kdc[14854](info): set up 2 sockets
Sep 23 20:21:22 dist-test-slave-wmd0 krb5kdc[14854](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 20:21:23.886981 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:21:23 dist-test-slave-wmd0 krb5kdc[14854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:24.062374 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:35539
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:35539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:24.445363 14867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:24.445744 14867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:24.445909 14867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:24.453522 14867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:24.453672 14867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:24.453756 14867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:24.453853 14867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:24.453920 14867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:24.464599 14867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:35539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:24.466382 14867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:24.468139 14867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:24.491081 14873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:24.491542 14875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:24.491645 14872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:24.492327 14867 server_base.cc:1034] running on GCE node
I20260923 20:21:24.493333 14867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:24.495012 14867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:24.496227 14867 hybrid_clock.cc:648] HybridClock initialized: now 1790194884496162 us; error 54 us; skew 500 ppm
I20260923 20:21:24.496635 14867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:24 dist-test-slave-wmd0 krb5kdc[14854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194884, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:24.504856 14867 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:24.507378 14867 webserver.cc:460] Webserver started at http://127.14.127.254:37047/ using document root <none> and password file <none>
I20260923 20:21:24.508587 14867 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:24.508800 14867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:24.509300 14867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:24.513793 14867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "64cad2d7ebb54e26a162f2274acc4e1c"
format_stamp: "Formatted at 2026-09-23 20:21:24 on dist-test-slave-wmd0"
server_key: "a50815f2f93bf4444785392d4bf78768"
server_key_iv: "2de7facd6a2941e19e5139bc9822950a"
server_key_version: "encryptionkey@0"
I20260923 20:21:24.514899 14867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "64cad2d7ebb54e26a162f2274acc4e1c"
format_stamp: "Formatted at 2026-09-23 20:21:24 on dist-test-slave-wmd0"
server_key: "a50815f2f93bf4444785392d4bf78768"
server_key_iv: "2de7facd6a2941e19e5139bc9822950a"
server_key_version: "encryptionkey@0"
I20260923 20:21:24.522091 14867 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260923 20:21:24.527251 14882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:24.529121 14867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:21:24.529323 14867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "64cad2d7ebb54e26a162f2274acc4e1c"
format_stamp: "Formatted at 2026-09-23 20:21:24 on dist-test-slave-wmd0"
server_key: "a50815f2f93bf4444785392d4bf78768"
server_key_iv: "2de7facd6a2941e19e5139bc9822950a"
server_key_version: "encryptionkey@0"
I20260923 20:21:24.529531 14867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:24.547731 14867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:24.553243 14867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:24.553664 14867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:24.579891 14867 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:35539
I20260923 20:21:24.579828 14933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:35539 every 8 connection(s)
I20260923 20:21:24.581837 14867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:24.587039 14934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:24.590294 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 14867
I20260923 20:21:24.590595 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:24.592545 14847 external_mini_cluster.cc:1428] Setting key 8f223fd8d311de6e6daf130761ddad42
I20260923 20:21:24.603669 14934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c: Bootstrap starting.
I20260923 20:21:24.608019 14934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:24.609771 14934 log.cc:824] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:24.613271 14934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c: No bootstrap required, opened a new log
I20260923 20:21:24.621199 14934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cad2d7ebb54e26a162f2274acc4e1c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35539 } }
Sep 23 20:21:24 dist-test-slave-wmd0 krb5kdc[14854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:24.621796 14934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:24.621989 14934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64cad2d7ebb54e26a162f2274acc4e1c, State: Initialized, Role: FOLLOWER
I20260923 20:21:24.622970 14934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cad2d7ebb54e26a162f2274acc4e1c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35539 } }
I20260923 20:21:24.623310 14934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:24.623495 14934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:24.623710 14934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:24.626348 14934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cad2d7ebb54e26a162f2274acc4e1c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35539 } }
I20260923 20:21:24.627105 14934 leader_election.cc:302] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64cad2d7ebb54e26a162f2274acc4e1c; no voters:
I20260923 20:21:24.628006 14934 leader_election.cc:288] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:24.628160 14939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:24.630220 14939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [term 1 LEADER]: Becoming Leader. State: Replica: 64cad2d7ebb54e26a162f2274acc4e1c, State: Running, Role: LEADER
I20260923 20:21:24.630997 14934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:24.631253 14939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cad2d7ebb54e26a162f2274acc4e1c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35539 } }
I20260923 20:21:24.633770 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:24.599918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.14.127.254:35539)
0923 20:21:24.600907 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:24.600927 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:24.600989 (+ 62us) server_negotiation.cc:407] Connection header received
0923 20:21:24.609425 (+ 8436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:24.609445 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:24.609989 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:24.610469 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:24.613387 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:24.614798 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:24.616401 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:24.617085 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:24.617306 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:24.623239 (+ 5933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:24.623251 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:24.623316 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:24.623364 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:24.626547 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:24.627551 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:24.627560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:24.627568 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:24.627811 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:24.628435 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:24.628440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:24.628444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:24.629363 (+ 919us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:24.629861 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:24.631233 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0923 20:21:24.631785 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":392,"threads_started":1}
I20260923 20:21:24.643064 14940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64cad2d7ebb54e26a162f2274acc4e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cad2d7ebb54e26a162f2274acc4e1c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35539 } } }
I20260923 20:21:24.643131 14941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64cad2d7ebb54e26a162f2274acc4e1c. Latest consensus state: current_term: 1 leader_uuid: "64cad2d7ebb54e26a162f2274acc4e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cad2d7ebb54e26a162f2274acc4e1c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35539 } } }
I20260923 20:21:24.643683 14941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:24.643700 14940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:24.644850 14949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:24.652710 14949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:24.665442 14949 catalog_manager.cc:1347] Generated new cluster ID: 91ea63b3bf674045b8d7ecb6de64153b
I20260923 20:21:24.665578 14949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:24.683805 14949 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:24.685885 14949 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:24.695700 14949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64cad2d7ebb54e26a162f2274acc4e1c: Generated new TSK 0
I20260923 20:21:24.696803 14949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:21:24.744308 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35539
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:25.105633 14961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:25.105943 14961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:25.106105 14961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:25.113351 14961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:25.113494 14961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:25.113677 14961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:21:25.123771 14961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:25.125008 14961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:25.126334 14961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:25.139900 14966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:25.139810 14967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:25.139853 14969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:25.140707 14961 server_base.cc:1034] running on GCE node
I20260923 20:21:25.141315 14961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:25.142184 14961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:25.143425 14961 hybrid_clock.cc:648] HybridClock initialized: now 1790194885143375 us; error 35 us; skew 500 ppm
I20260923 20:21:25.143741 14961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:25 dist-test-slave-wmd0 krb5kdc[14854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:25.146839 14961 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:21:25.148173 14961 webserver.cc:460] Webserver started at http://127.14.127.193:34015/ using document root <none> and password file <none>
I20260923 20:21:25.148842 14961 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:25.148978 14961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:25.149279 14961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:25.152143 14961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "dfc7c26d26944e6da39e435773b42642"
format_stamp: "Formatted at 2026-09-23 20:21:25 on dist-test-slave-wmd0"
server_key: "f86e8c0efd0e0ee05b87a812202537ae"
server_key_iv: "e3b85811a92ee0919587d7c2ff17997a"
server_key_version: "encryptionkey@0"
I20260923 20:21:25.152886 14961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "dfc7c26d26944e6da39e435773b42642"
format_stamp: "Formatted at 2026-09-23 20:21:25 on dist-test-slave-wmd0"
server_key: "f86e8c0efd0e0ee05b87a812202537ae"
server_key_iv: "e3b85811a92ee0919587d7c2ff17997a"
server_key_version: "encryptionkey@0"
I20260923 20:21:25.157979 14961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 20:21:25.161341 14976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:25.162526 14961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:21:25.162736 14961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "dfc7c26d26944e6da39e435773b42642"
format_stamp: "Formatted at 2026-09-23 20:21:25 on dist-test-slave-wmd0"
server_key: "f86e8c0efd0e0ee05b87a812202537ae"
server_key_iv: "e3b85811a92ee0919587d7c2ff17997a"
server_key_version: "encryptionkey@0"
I20260923 20:21:25.162923 14961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:25.179186 14961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:25.183053 14961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:25.183320 14961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:25.184214 14961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:25.185664 14961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:25.185762 14961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:25.185974 14961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:25.186053 14961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:25.224301 14961 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:40633
I20260923 20:21:25.224341 15088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:40633 every 8 connection(s)
I20260923 20:21:25.225965 14961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:21:25.231813 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 14961
I20260923 20:21:25.232034 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:21:25.232806 14847 external_mini_cluster.cc:1428] Setting key d244a624d72424ca71ad82380a0f1d84
Sep 23 20:21:25 dist-test-slave-wmd0 krb5kdc[14854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:25.246589 15091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:25.228907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:43891 (local address 127.14.127.254:35539)
0923 20:21:25.229352 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:25.229357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:25.230381 (+ 1024us) server_negotiation.cc:407] Connection header received
0923 20:21:25.232751 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:25.232755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:25.232824 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:25.232952 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:25.234813 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:25.235513 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:25.237086 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:25.237477 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:25.237619 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:25.240935 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:25.240940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:25.240961 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:25.240990 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:25.243560 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:25.244286 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:25.244294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:25.244297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:25.244470 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:25.244885 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:25.244890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:25.244893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:25.245142 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:25.245300 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:25.246023 (+ 723us) server_negotiation.cc:299] Negotiation successful
0923 20:21:25.246229 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":211,"threads_started":1}
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:21:25.247646 15092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:25.229523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35539 (local address 127.14.127.193:43891)
0923 20:21:25.230223 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:25.230264 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:25.232415 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:25.233201 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:25.233211 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:25.233799 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:25.234605 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:25.234672 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:25.235688 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:25.235693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:25.236925 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:25.236937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:25.237710 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:25.237717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:25.237969 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:25.238675 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:25.238705 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:25.240738 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:25.243697 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:25.243706 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:25.243723 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:25.244149 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:25.244582 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:25.244588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:25.244592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:25.244771 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:25.245274 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:25.245282 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:25.245664 (+ 382us) client_negotiation.cc:769] Sending connection context
0923 20:21:25.245925 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 20:21:25.246433 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":269,"threads_started":1}
I20260923 20:21:25.249132 15089 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35539
I20260923 20:21:25.249541 15089 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
I20260923 20:21:25.250455 15089 heartbeater.cc:511] Master 127.14.127.254:35539 requested a full tablet report, sending...
I20260923 20:21:25.253427 14899 ts_manager.cc:194] Registered new tserver with Master: dfc7c26d26944e6da39e435773b42642 (127.14.127.193:40633)
I20260923 20:21:25.255371 14899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:43891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:21:25.270064 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35539
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:25.638082 15095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:25.638365 15095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:25.638556 15095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:25.646870 15095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:25.647012 15095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:25.647202 15095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:21:25.658250 15095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:25.659572 15095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:25.661072 15095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:25.674772 15101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:25.674846 15100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:25.674897 15103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:25.675908 15095 server_base.cc:1034] running on GCE node
I20260923 20:21:25.676452 15095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:25.677289 15095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:25.678483 15095 hybrid_clock.cc:648] HybridClock initialized: now 1790194885678422 us; error 57 us; skew 500 ppm
I20260923 20:21:25.678829 15095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:25 dist-test-slave-wmd0 krb5kdc[14854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:25.681843 15095 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:21:25.683261 15095 webserver.cc:460] Webserver started at http://127.14.127.194:40107/ using document root <none> and password file <none>
I20260923 20:21:25.683933 15095 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:25.684090 15095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:25.684412 15095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:25.687438 15095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "8a505099273f4490bf9f6c0ab45486fe"
format_stamp: "Formatted at 2026-09-23 20:21:25 on dist-test-slave-wmd0"
server_key: "8a19528c935d1df98d61cfc8cbed2af6"
server_key_iv: "fab8b8e796cdad5b2ed861430bfb3cfd"
server_key_version: "encryptionkey@0"
I20260923 20:21:25.688284 15095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "8a505099273f4490bf9f6c0ab45486fe"
format_stamp: "Formatted at 2026-09-23 20:21:25 on dist-test-slave-wmd0"
server_key: "8a19528c935d1df98d61cfc8cbed2af6"
server_key_iv: "fab8b8e796cdad5b2ed861430bfb3cfd"
server_key_version: "encryptionkey@0"
I20260923 20:21:25.693380 15095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:21:25.698681 15110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:25.700353 15095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:21:25.700515 15095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "8a505099273f4490bf9f6c0ab45486fe"
format_stamp: "Formatted at 2026-09-23 20:21:25 on dist-test-slave-wmd0"
server_key: "8a19528c935d1df98d61cfc8cbed2af6"
server_key_iv: "fab8b8e796cdad5b2ed861430bfb3cfd"
server_key_version: "encryptionkey@0"
I20260923 20:21:25.700747 15095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:25.716360 15095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:25.720500 15095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:25.720737 15095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:25.721580 15095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:25.722846 15095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:25.722918 15095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:25.723007 15095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:25.723058 15095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:25.760991 15095 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:40551
I20260923 20:21:25.761042 15222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:40551 every 8 connection(s)
I20260923 20:21:25.762332 15095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:21:25.765609 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 15095
I20260923 20:21:25.765800 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:21:25.766377 14847 external_mini_cluster.cc:1428] Setting key a03378a6b97737d3a74be5e2e1c700dc
Sep 23 20:21:25 dist-test-slave-wmd0 krb5kdc[14854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
I20260923 20:21:25.788372 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:25.765409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:60453 (local address 127.14.127.254:35539)
0923 20:21:25.766419 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:25.766434 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:25.767874 (+ 1440us) server_negotiation.cc:407] Connection header received
0923 20:21:25.771083 (+ 3209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:25.771089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:25.771182 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:25.771301 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:25.773399 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:25.774402 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:25.776381 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:25.777022 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:25.777158 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:25.781362 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:25.781367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:25.781371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:25.781401 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:25.784217 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:25.785377 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:25.785385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:25.785391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:25.785555 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:25.786186 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:25.786194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:25.786200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:25.786587 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:25.786764 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:25.787577 (+ 813us) server_negotiation.cc:299] Negotiation successful
0923 20:21:25.787847 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":667,"threads_started":1}
I20260923 20:21:25.789615 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:25.765985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35539 (local address 127.14.127.194:60453)
0923 20:21:25.767664 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:25.767737 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:25.770814 (+ 3077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:25.771627 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:25.771635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:25.772326 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:25.773230 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:25.773250 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:25.774592 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:25.774596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:25.776230 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:25.776243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:25.777263 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:25.777271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:25.777500 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:25.778176 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:25.778198 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:25.781118 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:25.784426 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:25.784435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:25.784452 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:25.785134 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:25.785752 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:25.785756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:25.785758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:25.786056 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:25.786762 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:25.786768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:25.787190 (+ 422us) client_negotiation.cc:769] Sending connection context
0923 20:21:25.787479 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 20:21:25.787920 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"thread_start_us":815,"threads_started":1}
I20260923 20:21:25.791209 15223 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35539
I20260923 20:21:25.791518 15223 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:25.792246 15223 heartbeater.cc:511] Master 127.14.127.254:35539 requested a full tablet report, sending...
I20260923 20:21:25.793715 14899 ts_manager.cc:194] Registered new tserver with Master: 8a505099273f4490bf9f6c0ab45486fe (127.14.127.194:40551)
I20260923 20:21:25.794687 14899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:60453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:21:25.806061 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35539
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:26.169104 15229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:26.169590 15229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:26.169773 15229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:26.176951 15229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:26.177100 15229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:26.177287 15229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:21:26.187170 15229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35539
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:26.188496 15229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:26.189826 15229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:26.202910 15235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:26.202883 15237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:26.202939 15234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:26.203679 15229 server_base.cc:1034] running on GCE node
I20260923 20:21:26.204344 15229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:26.205368 15229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:26.206660 15229 hybrid_clock.cc:648] HybridClock initialized: now 1790194886206567 us; error 53 us; skew 500 ppm
I20260923 20:21:26.206966 15229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[14854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:26.209918 15229 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:21:26.211617 15229 webserver.cc:460] Webserver started at http://127.14.127.195:38209/ using document root <none> and password file <none>
I20260923 20:21:26.212304 15229 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:26.212453 15229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:26.212751 15229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:26.215624 15229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "7856203066ff4ad88fb9f248ba4b0996"
format_stamp: "Formatted at 2026-09-23 20:21:26 on dist-test-slave-wmd0"
server_key: "c09a7e7818550d448514e6c8943d5319"
server_key_iv: "ec7f63ea8c344502b86c1f29155e2aa7"
server_key_version: "encryptionkey@0"
I20260923 20:21:26.216387 15229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "7856203066ff4ad88fb9f248ba4b0996"
format_stamp: "Formatted at 2026-09-23 20:21:26 on dist-test-slave-wmd0"
server_key: "c09a7e7818550d448514e6c8943d5319"
server_key_iv: "ec7f63ea8c344502b86c1f29155e2aa7"
server_key_version: "encryptionkey@0"
I20260923 20:21:26.221463 15229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:21:26.225093 15244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:26.226701 15229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:26.226858 15229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "7856203066ff4ad88fb9f248ba4b0996"
format_stamp: "Formatted at 2026-09-23 20:21:26 on dist-test-slave-wmd0"
server_key: "c09a7e7818550d448514e6c8943d5319"
server_key_iv: "ec7f63ea8c344502b86c1f29155e2aa7"
server_key_version: "encryptionkey@0"
I20260923 20:21:26.227066 15229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:26.247892 15229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:26.251806 15229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:26.252071 15229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:26.252964 15229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:26.254215 15229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:26.254292 15229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:26.254371 15229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:26.254415 15229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:26.258474 15089 heartbeater.cc:503] Master 127.14.127.254:35539 was elected leader, sending a full tablet report...
I20260923 20:21:26.296906 15229 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:44665
I20260923 20:21:26.297217 15356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:44665 every 8 connection(s)
I20260923 20:21:26.298303 15229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:21:26.302284 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 15229
I20260923 20:21:26.302477 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:21:26.302956 14847 external_mini_cluster.cc:1428] Setting key eab05452327f276eaf3ecce2be177933
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[14854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:26.318799 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:26.301295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:39059 (local address 127.14.127.254:35539)
0923 20:21:26.302215 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:26.302222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:26.303441 (+ 1219us) server_negotiation.cc:407] Connection header received
0923 20:21:26.305504 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:26.305510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:26.305718 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:26.305814 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:26.307613 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.308393 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.309829 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.310251 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.310344 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:26.313538 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:26.313542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:26.313545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:26.313586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:26.316031 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.316675 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.316679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.316682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.316790 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.317135 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.317138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.317140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.317381 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:26.317544 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:26.318159 (+ 615us) server_negotiation.cc:299] Negotiation successful
0923 20:21:26.318373 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":514,"threads_started":1}
I20260923 20:21:26.319869 15360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:26.302202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35539 (local address 127.14.127.195:39059)
0923 20:21:26.303255 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:26.303309 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:26.305252 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:26.306135 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:26.306147 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:26.306893 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:26.307455 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:26.307472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.308515 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.308518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:26.309706 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:26.309717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.310421 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.310425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:26.310605 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:26.311321 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:26.311342 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:26.313375 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:26.316157 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:26.316163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:26.316174 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:26.316567 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:26.316876 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:26.316878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:26.316881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:26.317040 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:26.317510 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:26.317515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:26.317805 (+ 290us) client_negotiation.cc:769] Sending connection context
0923 20:21:26.318019 (+ 214us) client_negotiation.cc:240] Negotiation successful
0923 20:21:26.318550 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":524,"threads_started":1}
I20260923 20:21:26.321040 15357 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35539
I20260923 20:21:26.321295 15357 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:26.321910 15357 heartbeater.cc:511] Master 127.14.127.254:35539 requested a full tablet report, sending...
I20260923 20:21:26.323182 14899 ts_manager.cc:194] Registered new tserver with Master: 7856203066ff4ad88fb9f248ba4b0996 (127.14.127.195:44665)
I20260923 20:21:26.323949 14899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:39059
I20260923 20:21:26.331874 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[14854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194886, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[14854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194886, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:21:26.375507 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:26.360548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.14.127.193:40633)
0923 20:21:26.361220 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:26.361230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:26.361262 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:21:26.361431 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:26.361440 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:26.361660 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:26.361821 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:26.363321 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.364309 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.365546 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.366172 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.366315 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:26.370188 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:26.370198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:26.370211 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:26.370246 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:26.372884 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.373500 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.373507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.373509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.373632 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.373987 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.373993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.373996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.374320 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:26.374671 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:26.374908 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 20:21:26.375168 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":322,"threads_started":1}
W20260923 20:21:26.376803 15004 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41330
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[14854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[14854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:21:26.417784 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:26.404623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.14.127.193:40633)
0923 20:21:26.404856 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:26.404862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:26.405030 (+ 168us) server_negotiation.cc:407] Connection header received
0923 20:21:26.405241 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:26.405252 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:26.405317 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:26.405438 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:26.406838 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.407524 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.408761 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.409200 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.409327 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:26.413029 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:26.413034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:26.413037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:26.413072 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:26.415396 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.415982 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.415989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.415992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.416123 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.416465 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.416471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.416474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.416720 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:26.416955 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:26.417250 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 20:21:26.417462 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 20:21:26.419371 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 14961
I20260923 20:21:26.437335 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:26.423117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.14.127.193:40633)
0923 20:21:26.423328 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:26.423334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:26.423547 (+ 213us) server_negotiation.cc:407] Connection header received
0923 20:21:26.423725 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:26.423731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:26.423793 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:26.423886 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:26.424997 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.425682 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.427421 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.427811 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.427940 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:26.431258 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:26.431264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:26.431269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:26.431313 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:26.435197 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.435798 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.435802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.435804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.436020 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.436312 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.436314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.436317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.436549 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:26.436728 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:26.436924 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 20:21:26.437071 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:21:26.438027 15084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:26.523388 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 14961
I20260923 20:21:26.539923 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 15095
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[14854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:21:26.555275 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:26.540844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.14.127.194:40551)
0923 20:21:26.541403 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:26.541409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:26.541434 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:21:26.541493 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:26.541499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:26.541705 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:26.541834 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:26.543027 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.544033 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.545230 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.545846 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.545939 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:26.549752 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:26.549763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:26.549777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:26.549812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:26.552737 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.553299 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.553305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.553308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.553432 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.553798 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.553801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.553803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.554117 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:26.554343 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:26.554585 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 20:21:26.554977 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
I20260923 20:21:26.556550 15218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:26.639039 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 15095
I20260923 20:21:26.655087 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 15229
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[14854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:21:26.670795 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:26.656160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.14.127.195:44665)
0923 20:21:26.656814 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:26.656822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:26.656851 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:21:26.656909 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:26.656915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:26.657182 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:26.657311 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:26.658694 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.659719 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.660934 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:26.661470 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:26.661565 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:26.665526 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:26.665536 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:26.665550 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:26.665585 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:26.668260 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.668849 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.668855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.668857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.668970 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:26.669264 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:26.669267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:26.669269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:26.669584 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:26.669808 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:26.670035 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 20:21:26.670225 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":343,"threads_started":1}
I20260923 20:21:26.672056 15352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:26.753890 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 15229
I20260923 20:21:26.769419 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 14867
I20260923 20:21:26.770566 14929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:26.871683 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 14867
2026-09-23T20:21:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4082 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[15379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[15379](info): set up 2 sockets
Sep 23 20:21:26 dist-test-slave-wmd0 krb5kdc[15379](info): commencing operation
krb5kdc: starting...
W20260923 20:21:27.958664 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:21:28 dist-test-slave-wmd0 krb5kdc[15379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:28.134203 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:40525
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:40525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:28.450344 15392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:28.450707 15392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:28.450814 15392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:28.458575 15392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:28.458699 15392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:28.458757 15392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:28.458814 15392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:28.458858 15392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:28.469224 15392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:40525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:40525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:28.470515 15392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:28.471817 15392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:28.483445 15397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:28.483472 15398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:28.483518 15400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:28.484587 15392 server_base.cc:1034] running on GCE node
I20260923 20:21:28.485415 15392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:28.486931 15392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:28.488147 15392 hybrid_clock.cc:648] HybridClock initialized: now 1790194888488086 us; error 53 us; skew 500 ppm
I20260923 20:21:28.488521 15392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:28 dist-test-slave-wmd0 krb5kdc[15379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194888, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:28.491601 15392 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:28.493059 15392 webserver.cc:460] Webserver started at http://127.14.127.254:33543/ using document root <none> and password file <none>
I20260923 20:21:28.493693 15392 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:28.493790 15392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:28.494050 15392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:28.497015 15392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "57a1f2a09644435fb86b3295f295473f"
format_stamp: "Formatted at 2026-09-23 20:21:28 on dist-test-slave-wmd0"
server_key: "2a7350cb40804d6fa860e68f9ee543b6"
server_key_iv: "2d18e8d1b3fa1d0b094286b69e0d8046"
server_key_version: "encryptionkey@0"
I20260923 20:21:28.497712 15392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "57a1f2a09644435fb86b3295f295473f"
format_stamp: "Formatted at 2026-09-23 20:21:28 on dist-test-slave-wmd0"
server_key: "2a7350cb40804d6fa860e68f9ee543b6"
server_key_iv: "2d18e8d1b3fa1d0b094286b69e0d8046"
server_key_version: "encryptionkey@0"
I20260923 20:21:28.502956 15392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:21:28.506314 15407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:28.507678 15392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:28.507862 15392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "57a1f2a09644435fb86b3295f295473f"
format_stamp: "Formatted at 2026-09-23 20:21:28 on dist-test-slave-wmd0"
server_key: "2a7350cb40804d6fa860e68f9ee543b6"
server_key_iv: "2d18e8d1b3fa1d0b094286b69e0d8046"
server_key_version: "encryptionkey@0"
I20260923 20:21:28.508075 15392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:28.533038 15392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:28.537478 15392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:28.537772 15392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:28.556195 15392 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:40525
I20260923 20:21:28.556223 15458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:40525 every 8 connection(s)
I20260923 20:21:28.557539 15392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:28.560452 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 15392
I20260923 20:21:28.560712 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:28.561267 15459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:28.561393 14847 external_mini_cluster.cc:1428] Setting key 00597ae16aaa6745824acca5b4cf699c
I20260923 20:21:28.574203 15459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f: Bootstrap starting.
Sep 23 20:21:28 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:28.577739 15459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:28.578866 15459 log.cc:824] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:28.581723 15459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f: No bootstrap required, opened a new log
I20260923 20:21:28.584443 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:28.563390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.14.127.254:40525)
0923 20:21:28.564143 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:28.564158 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:28.564201 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:21:28.566067 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:28.566088 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:28.566458 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:28.566904 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:28.568341 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:28.569592 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:28.570885 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:28.571609 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:28.571844 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:28.576721 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:28.576729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:28.576743 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:28.576778 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:28.580409 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:28.581083 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:28.581089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:28.581095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:28.581256 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:28.581648 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:28.581652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:28.581654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:28.582204 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:28.582511 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:28.583059 (+ 548us) server_negotiation.cc:299] Negotiation successful
0923 20:21:28.583468 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":337,"threads_started":1}
I20260923 20:21:28.589948 15459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1f2a09644435fb86b3295f295473f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40525 } }
I20260923 20:21:28.590324 15459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:28.590426 15459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a1f2a09644435fb86b3295f295473f, State: Initialized, Role: FOLLOWER
I20260923 20:21:28.591053 15459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1f2a09644435fb86b3295f295473f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40525 } }
I20260923 20:21:28.591296 15459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:28.591442 15459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:28.591619 15459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:28.593931 15459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1f2a09644435fb86b3295f295473f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40525 } }
I20260923 20:21:28.594403 15459 leader_election.cc:302] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a1f2a09644435fb86b3295f295473f; no voters:
I20260923 20:21:28.595070 15459 leader_election.cc:288] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:28.595196 15464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:28.595866 15464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [term 1 LEADER]: Becoming Leader. State: Replica: 57a1f2a09644435fb86b3295f295473f, State: Running, Role: LEADER
I20260923 20:21:28.596390 15464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1f2a09644435fb86b3295f295473f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40525 } }
I20260923 20:21:28.597162 15459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:28.599598 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a1f2a09644435fb86b3295f295473f. Latest consensus state: current_term: 1 leader_uuid: "57a1f2a09644435fb86b3295f295473f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1f2a09644435fb86b3295f295473f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40525 } } }
I20260923 20:21:28.599956 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:28.600672 15465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a1f2a09644435fb86b3295f295473f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1f2a09644435fb86b3295f295473f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40525 } } }
I20260923 20:21:28.601027 15465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:28.603776 15472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:28.608058 15472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:28.616251 15472 catalog_manager.cc:1347] Generated new cluster ID: 49b14d6d591c440ca29436847b031dbc
I20260923 20:21:28.616361 15472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:28.634421 15472 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:28.636279 15472 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:28.646755 15472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57a1f2a09644435fb86b3295f295473f: Generated new TSK 0
I20260923 20:21:28.647810 15472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:21:28.690419 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:40525
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:29.007200 15485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:29.007499 15485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:29.007685 15485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:29.015425 15485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:29.015519 15485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:29.015720 15485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:21:29.025259 15485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:40525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:29.026552 15485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:29.027886 15485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:29.041087 15490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:29.041064 15491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:29.041145 15493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:29.041975 15485 server_base.cc:1034] running on GCE node
I20260923 20:21:29.042665 15485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:29.043594 15485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:29.044792 15485 hybrid_clock.cc:648] HybridClock initialized: now 1790194889044744 us; error 48 us; skew 500 ppm
I20260923 20:21:29.045116 15485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:29 dist-test-slave-wmd0 krb5kdc[15379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:29.048236 15485 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:21:29.049749 15485 webserver.cc:460] Webserver started at http://127.14.127.193:40401/ using document root <none> and password file <none>
I20260923 20:21:29.050439 15485 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:29.050586 15485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:29.050911 15485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:29.053694 15485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "68f2adbf738944788a693a70aba782e8"
format_stamp: "Formatted at 2026-09-23 20:21:29 on dist-test-slave-wmd0"
server_key: "9437a5479c2eed369227e1a95def250b"
server_key_iv: "7b2741beba933d389976eca32be978fb"
server_key_version: "encryptionkey@0"
I20260923 20:21:29.054509 15485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "68f2adbf738944788a693a70aba782e8"
format_stamp: "Formatted at 2026-09-23 20:21:29 on dist-test-slave-wmd0"
server_key: "9437a5479c2eed369227e1a95def250b"
server_key_iv: "7b2741beba933d389976eca32be978fb"
server_key_version: "encryptionkey@0"
I20260923 20:21:29.059496 15485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:21:29.063040 15500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:29.064354 15485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:21:29.064527 15485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "68f2adbf738944788a693a70aba782e8"
format_stamp: "Formatted at 2026-09-23 20:21:29 on dist-test-slave-wmd0"
server_key: "9437a5479c2eed369227e1a95def250b"
server_key_iv: "7b2741beba933d389976eca32be978fb"
server_key_version: "encryptionkey@0"
I20260923 20:21:29.064725 15485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:29.085273 15485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:29.089169 15485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:29.089458 15485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:29.090372 15485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:29.091760 15485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:29.091869 15485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:29.092028 15485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:29.092101 15485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:29.132354 15485 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:36665
I20260923 20:21:29.132431 15612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:36665 every 8 connection(s)
I20260923 20:21:29.134110 15485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:21:29.137879 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 15485
I20260923 20:21:29.138111 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:21:29.139025 14847 external_mini_cluster.cc:1428] Setting key be1d8f6db604c71cb80dcb8377c50f21
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Sep 23 20:21:29 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:29.168779 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:29.137022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:57603 (local address 127.14.127.254:40525)
0923 20:21:29.142954 (+ 5932us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:29.142963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:29.147427 (+ 4464us) server_negotiation.cc:407] Connection header received
0923 20:21:29.150213 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:29.150220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:29.150327 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:29.150461 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:29.152619 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:29.153914 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:29.156403 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:29.156817 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:29.156958 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:29.162602 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:29.162666 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:29.162675 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:29.162718 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:29.165394 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:29.166160 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:29.166169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:29.166176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:29.166321 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:29.166757 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:29.166765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:29.166771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:29.167151 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:29.167299 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:29.168000 (+ 701us) server_negotiation.cc:299] Negotiation successful
0923 20:21:29.168253 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5787,"thread_start_us":5697,"threads_started":1}
I20260923 20:21:29.169647 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:29.137804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:40525 (local address 127.14.127.193:57603)
0923 20:21:29.147211 (+ 9407us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:29.147266 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:29.149906 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:29.150743 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:29.150758 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:29.151603 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:29.152431 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:29.152459 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:29.154140 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:29.154146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:29.156248 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:29.156261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:29.157056 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:29.157063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:29.157314 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:29.158025 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:29.158051 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:29.162350 (+ 4299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:29.165533 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:29.165543 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:29.165568 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:29.166023 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:29.166425 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:29.166428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:29.166430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:29.166605 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:29.167277 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:29.167283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:29.167614 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 20:21:29.167837 (+ 223us) client_negotiation.cc:240] Negotiation successful
0923 20:21:29.168332 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9100,"thread_start_us":8849,"threads_started":1}
I20260923 20:21:29.171386 15613 heartbeater.cc:347] Connected to a master server at 127.14.127.254:40525
I20260923 20:21:29.171739 15613 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:29.172698 15613 heartbeater.cc:511] Master 127.14.127.254:40525 requested a full tablet report, sending...
I20260923 20:21:29.175029 15424 ts_manager.cc:194] Registered new tserver with Master: 68f2adbf738944788a693a70aba782e8 (127.14.127.193:36665)
I20260923 20:21:29.176512 15424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:57603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:21:29.184106 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:40525
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:29.493741 15619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:29.493999 15619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:29.494158 15619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:29.501621 15619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:29.501719 15619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:29.501943 15619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:21:29.511968 15619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:40525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:29.513293 15619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:29.514599 15619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:29.527498 15627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:29.527561 15625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:29.527824 15624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:29.528878 15619 server_base.cc:1034] running on GCE node
I20260923 20:21:29.529471 15619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:29.530370 15619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:29.531610 15619 hybrid_clock.cc:648] HybridClock initialized: now 1790194889531544 us; error 81 us; skew 500 ppm
I20260923 20:21:29.531941 15619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:29 dist-test-slave-wmd0 krb5kdc[15379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:29.538992 15619 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:21:29.540493 15619 webserver.cc:460] Webserver started at http://127.14.127.194:35779/ using document root <none> and password file <none>
I20260923 20:21:29.541200 15619 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:29.541349 15619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:29.541671 15619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:29.544538 15619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "c9a8a46156104231b6421be440cbb31d"
format_stamp: "Formatted at 2026-09-23 20:21:29 on dist-test-slave-wmd0"
server_key: "d443f1ad763cad7d3aff1211c38ac268"
server_key_iv: "101a114cc864f7b1043074c0024d58d7"
server_key_version: "encryptionkey@0"
I20260923 20:21:29.545312 15619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "c9a8a46156104231b6421be440cbb31d"
format_stamp: "Formatted at 2026-09-23 20:21:29 on dist-test-slave-wmd0"
server_key: "d443f1ad763cad7d3aff1211c38ac268"
server_key_iv: "101a114cc864f7b1043074c0024d58d7"
server_key_version: "encryptionkey@0"
I20260923 20:21:29.550292 15619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:21:29.553696 15634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:29.555147 15619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 20:21:29.555312 15619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "c9a8a46156104231b6421be440cbb31d"
format_stamp: "Formatted at 2026-09-23 20:21:29 on dist-test-slave-wmd0"
server_key: "d443f1ad763cad7d3aff1211c38ac268"
server_key_iv: "101a114cc864f7b1043074c0024d58d7"
server_key_version: "encryptionkey@0"
I20260923 20:21:29.555510 15619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:29.585227 15619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:29.589666 15619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:29.589963 15619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:29.590965 15619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:29.592231 15619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:29.592331 15619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:29.592475 15619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:29.592556 15619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:29.626524 15619 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:46601
I20260923 20:21:29.626577 15746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:46601 every 8 connection(s)
I20260923 20:21:29.627831 15619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:21:29.629403 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 15619
I20260923 20:21:29.629593 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:21:29.630144 14847 external_mini_cluster.cc:1428] Setting key fe69db875c16875710d5383be9a0e842
Sep 23 20:21:29 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194889, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
I20260923 20:21:29.647903 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:29.630667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:33871 (local address 127.14.127.254:40525)
0923 20:21:29.630842 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:29.630847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:29.632235 (+ 1388us) server_negotiation.cc:407] Connection header received
0923 20:21:29.633989 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:29.633994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:29.634086 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:29.634188 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:29.635722 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:29.636659 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:29.638154 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:29.638559 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:29.638736 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:29.642240 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:29.642245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:29.642249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:29.642278 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:29.644817 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:29.645549 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:29.645555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:29.645567 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:29.645719 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:29.646145 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:29.646148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:29.646151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:29.646396 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:29.646503 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:29.647219 (+ 716us) server_negotiation.cc:299] Negotiation successful
0923 20:21:29.647486 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:21:29.648955 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:29.631288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:40525 (local address 127.14.127.194:33871)
0923 20:21:29.632081 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:29.632127 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:29.633742 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:29.634423 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:29.634431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:29.635018 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:29.635554 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:29.635580 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:29.636824 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:29.636829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:29.638013 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:29.638024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:29.638818 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:29.638825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:29.639057 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:29.639721 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:29.639741 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:29.642018 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:29.644997 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:29.645005 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:29.645028 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:29.645421 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:29.645852 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:29.645856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:29.645858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:29.646039 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:29.646498 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:29.646503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:29.646877 (+ 374us) client_negotiation.cc:769] Sending connection context
0923 20:21:29.647101 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 20:21:29.647586 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":304,"threads_started":1}
I20260923 20:21:29.650374 15747 heartbeater.cc:347] Connected to a master server at 127.14.127.254:40525
I20260923 20:21:29.650731 15747 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:29.651412 15747 heartbeater.cc:511] Master 127.14.127.254:40525 requested a full tablet report, sending...
I20260923 20:21:29.652817 15424 ts_manager.cc:194] Registered new tserver with Master: c9a8a46156104231b6421be440cbb31d (127.14.127.194:46601)
I20260923 20:21:29.653609 15424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:33871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:21:29.669097 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:40525
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:29.977917 15752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:29.978178 15752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:29.978273 15752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:29.985278 15752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:29.985375 15752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:29.985558 15752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:21:29.994971 15752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:40525
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:29.996165 15752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:29.997493 15752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:30.010859 15757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:30.010903 15758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:30.010941 15760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:30.011541 15752 server_base.cc:1034] running on GCE node
I20260923 20:21:30.012231 15752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:30.013070 15752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:30.014240 15752 hybrid_clock.cc:648] HybridClock initialized: now 1790194890014169 us; error 49 us; skew 500 ppm
I20260923 20:21:30.014689 15752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194890, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:30.019130 15752 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:21:30.021350 15752 webserver.cc:460] Webserver started at http://127.14.127.195:33457/ using document root <none> and password file <none>
I20260923 20:21:30.022202 15752 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:30.022351 15752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:30.022758 15752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:30.025751 15752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "2793939e389a4471b66a58c1cff3658e"
format_stamp: "Formatted at 2026-09-23 20:21:30 on dist-test-slave-wmd0"
server_key: "7486e28397157c5122251117bb8274cc"
server_key_iv: "be8558b00a516b70fe510f3dd02c4e05"
server_key_version: "encryptionkey@0"
I20260923 20:21:30.026475 15752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "2793939e389a4471b66a58c1cff3658e"
format_stamp: "Formatted at 2026-09-23 20:21:30 on dist-test-slave-wmd0"
server_key: "7486e28397157c5122251117bb8274cc"
server_key_iv: "be8558b00a516b70fe510f3dd02c4e05"
server_key_version: "encryptionkey@0"
I20260923 20:21:30.031275 15752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:21:30.034549 15767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:30.035874 15752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:21:30.036032 15752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "2793939e389a4471b66a58c1cff3658e"
format_stamp: "Formatted at 2026-09-23 20:21:30 on dist-test-slave-wmd0"
server_key: "7486e28397157c5122251117bb8274cc"
server_key_iv: "be8558b00a516b70fe510f3dd02c4e05"
server_key_version: "encryptionkey@0"
I20260923 20:21:30.036264 15752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:30.053447 15752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:30.057437 15752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:30.057711 15752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:30.058800 15752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:30.060266 15752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:30.060340 15752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:30.060424 15752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:30.060485 15752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:30.094076 15752 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:43413
I20260923 20:21:30.094162 15879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:43413 every 8 connection(s)
I20260923 20:21:30.095428 15752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:21:30.104087 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 15752
I20260923 20:21:30.104297 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:21:30.104705 14847 external_mini_cluster.cc:1428] Setting key 5eacc8a9bd3f567b080f3b3d91a85ee6
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194890, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:30.114705 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.097874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:54479 (local address 127.14.127.254:40525)
0923 20:21:30.098050 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.098055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.099324 (+ 1269us) server_negotiation.cc:407] Connection header received
0923 20:21:30.101161 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.101166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.101272 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.101385 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:30.103038 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.103785 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.105456 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.105925 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.106055 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.109406 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:30.109411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:30.109414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:30.109445 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:30.112046 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.112812 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.112816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.112818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.112930 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.113290 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.113293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.113295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.113536 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:30.113642 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.114245 (+ 603us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.114398 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:21:30.115618 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.098434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:40525 (local address 127.14.127.195:54479)
0923 20:21:30.099166 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:30.099208 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:30.100889 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:30.101624 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:30.101632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:30.102203 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:30.102875 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.102895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.103940 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.103945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.105316 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.105327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.106146 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.106155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.106385 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.107079 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:30.107106 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:30.109202 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:30.112191 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:30.112197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:30.112209 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:30.112673 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:30.113018 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:30.113021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:30.113024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:30.113193 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:30.113641 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:30.113647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:30.113940 (+ 293us) client_negotiation.cc:769] Sending connection context
0923 20:21:30.114153 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 20:21:30.114506 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":320,"threads_started":1}
I20260923 20:21:30.116904 15880 heartbeater.cc:347] Connected to a master server at 127.14.127.254:40525
I20260923 20:21:30.117188 15880 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:30.117957 15880 heartbeater.cc:511] Master 127.14.127.254:40525 requested a full tablet report, sending...
I20260923 20:21:30.119493 15424 ts_manager.cc:194] Registered new tserver with Master: 2793939e389a4471b66a58c1cff3658e (127.14.127.195:43413)
I20260923 20:21:30.120131 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:21:30.120425 15424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:54479
I20260923 20:21:30.142454 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.129765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.14.127.254:40525)
0923 20:21:30.129917 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.129921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.130397 (+ 476us) server_negotiation.cc:407] Connection header received
0923 20:21:30.130519 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.130523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.130579 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.130752 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:30.132245 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.133365 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.134648 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.135316 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.135428 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.137654 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:30.137659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:30.137662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:30.137691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:30.140187 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.140818 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.140822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.140824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.140939 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.141338 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.141342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.141344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.141657 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:30.141819 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.142098 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.142242 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:21:30.151340 15424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:21:30.158034 15424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:30.179522 15613 heartbeater.cc:503] Master 127.14.127.254:40525 was elected leader, sending a full tablet report...
I20260923 20:21:30.181746 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.171238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:43413 (local address 127.0.0.1:44810)
0923 20:21:30.172041 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:30.172080 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:30.172297 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:30.173721 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:30.173728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:30.173768 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:30.174273 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.174290 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.177073 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.177078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.178884 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.178899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.180656 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.180664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.180862 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.180924 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 20:21:30.181030 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:21:30.181239 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":27,"thread_start_us":348,"threads_started":1}
I20260923 20:21:30.182070 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.172656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.14.127.194:46601)
0923 20:21:30.173168 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.173177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.173204 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:21:30.173306 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.173315 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.174085 (+ 770us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.174882 (+ 797us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:30.175061 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.176010 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.179315 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.180711 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.180870 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.181051 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.181608 (+ 557us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.181791 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260923 20:21:30.182076 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.172391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:46601 (local address 127.0.0.1:34130)
0923 20:21:30.172881 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:30.172901 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:30.173007 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:30.174302 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:30.174307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:30.174334 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:30.174725 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.174734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.176139 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.176145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.179162 (+ 3017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.179176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.181204 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.181211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.181397 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.181439 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:21:30.181533 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 20:21:30.181669 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":7,"thread_start_us":218,"threads_started":1}
I20260923 20:21:30.183534 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.171486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.14.127.195:43413)
0923 20:21:30.172229 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.172239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.172273 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:21:30.173191 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.173199 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.173445 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.174529 (+ 1084us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:30.174895 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.176925 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.179038 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.180396 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.182867 (+ 2471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.183027 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.183096 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.183243 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":417,"threads_started":1}
I20260923 20:21:30.184983 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.173009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36665 (local address 127.0.0.1:38590)
0923 20:21:30.173525 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:30.173564 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:30.173678 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:30.177156 (+ 3478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:30.177163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:30.177191 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:30.177681 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.177692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.179594 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.179600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.182644 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.182662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.184198 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.184204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.184372 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.184404 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:21:30.184498 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 20:21:30.184619 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":3,"thread_start_us":775,"threads_started":1}
I20260923 20:21:30.185039 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.173609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.14.127.193:36665)
0923 20:21:30.176618 (+ 3009us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.176626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.176657 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:21:30.176758 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.176765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.176975 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.177172 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:30.177859 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.179428 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.182800 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.183992 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.184195 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.184351 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.184566 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.184722 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2844,"thread_start_us":2698,"threads_started":1}
I20260923 20:21:30.186484 15682 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4261a94a734a2b932502c6ce126096 (DEFAULT_TABLE table=test-table [id=c88c8d6c12e64c66a9b8140cf5dcf17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:30.187103 15815 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4261a94a734a2b932502c6ce126096 (DEFAULT_TABLE table=test-table [id=c88c8d6c12e64c66a9b8140cf5dcf17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:30.188035 15682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4261a94a734a2b932502c6ce126096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:30.188488 15815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4261a94a734a2b932502c6ce126096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:30.188251 15548 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4261a94a734a2b932502c6ce126096 (DEFAULT_TABLE table=test-table [id=c88c8d6c12e64c66a9b8140cf5dcf17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:30.189317 15548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4261a94a734a2b932502c6ce126096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:30.200438 15898 tablet_bootstrap.cc:492] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8: Bootstrap starting.
I20260923 20:21:30.204448 15899 tablet_bootstrap.cc:492] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d: Bootstrap starting.
I20260923 20:21:30.206359 15900 tablet_bootstrap.cc:492] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e: Bootstrap starting.
I20260923 20:21:30.206387 15898 tablet_bootstrap.cc:654] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:30.207492 15898 log.cc:824] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:30.207877 15899 tablet_bootstrap.cc:654] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:30.209161 15899 log.cc:824] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:30.210048 15900 tablet_bootstrap.cc:654] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:30.211530 15898 tablet_bootstrap.cc:492] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8: No bootstrap required, opened a new log
I20260923 20:21:30.211825 15898 ts_tablet_manager.cc:1397] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260923 20:21:30.211846 15900 log.cc:824] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:30.212984 15899 tablet_bootstrap.cc:492] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d: No bootstrap required, opened a new log
I20260923 20:21:30.213318 15899 ts_tablet_manager.cc:1397] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 20:21:30.215849 15900 tablet_bootstrap.cc:492] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e: No bootstrap required, opened a new log
I20260923 20:21:30.216084 15900 ts_tablet_manager.cc:1397] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:21:30.220427 15899 raft_consensus.cc:348] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.220782 15899 raft_consensus.cc:374] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:30.220882 15899 raft_consensus.cc:729] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a8a46156104231b6421be440cbb31d, State: Initialized, Role: FOLLOWER
I20260923 20:21:30.221582 15899 consensus_queue.cc:260] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.222882 15747 heartbeater.cc:503] Master 127.14.127.254:40525 was elected leader, sending a full tablet report...
I20260923 20:21:30.223093 15898 raft_consensus.cc:348] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.223495 15898 raft_consensus.cc:374] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:30.223521 15899 ts_tablet_manager.cc:1428] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260923 20:21:30.223580 15898 raft_consensus.cc:729] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f2adbf738944788a693a70aba782e8, State: Initialized, Role: FOLLOWER
I20260923 20:21:30.224112 15898 consensus_queue.cc:260] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.225900 15898 ts_tablet_manager.cc:1428] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 20:21:30.226770 15900 raft_consensus.cc:348] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.227229 15900 raft_consensus.cc:374] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:30.227356 15900 raft_consensus.cc:729] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2793939e389a4471b66a58c1cff3658e, State: Initialized, Role: FOLLOWER
I20260923 20:21:30.228065 15900 consensus_queue.cc:260] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.229673 15900 ts_tablet_manager.cc:1428] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260923 20:21:30.230101 15880 heartbeater.cc:503] Master 127.14.127.254:40525 was elected leader, sending a full tablet report...
I20260923 20:21:30.279631 15905 raft_consensus.cc:482] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:30.279938 15905 raft_consensus.cc:504] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.282181 15905 leader_election.cc:288] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2793939e389a4471b66a58c1cff3658e (127.14.127.195:43413), c9a8a46156104231b6421be440cbb31d (127.14.127.194:46601)
I20260923 20:21:30.288636 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.282599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:59191 (local address 127.14.127.195:43413)
0923 20:21:30.282865 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.282872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.283292 (+ 420us) server_negotiation.cc:407] Connection header received
0923 20:21:30.283561 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.283571 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.283683 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.283780 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:30.284386 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.285108 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.286914 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.287693 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.287870 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.287948 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.288225 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.288354 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:21:30.288763 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.282521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:43413 (local address 127.14.127.193:59191)
0923 20:21:30.283176 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:30.283196 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:30.283368 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:30.283857 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:30.283862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:30.283897 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:30.284243 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.284252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.285237 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.285245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.286766 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.286778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.287908 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.287914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.288064 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.288088 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:21:30.288198 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:21:30.288327 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":1,"thread_start_us":394,"threads_started":1}
I20260923 20:21:30.289105 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.283630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:32973 (local address 127.14.127.194:46601)
0923 20:21:30.283848 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.283853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.284052 (+ 199us) server_negotiation.cc:407] Connection header received
0923 20:21:30.284267 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.284273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.284341 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.284438 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:30.284942 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.285628 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.287404 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.288160 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.288338 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.288413 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.288702 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.288825 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:21:30.289659 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.283333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:46601 (local address 127.14.127.193:32973)
0923 20:21:30.283955 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:30.283971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:30.284126 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:30.284452 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:30.284456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:30.284478 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:30.284801 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.284809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.285776 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.285781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.287230 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:30.287242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.288380 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.288386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:30.288534 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.288562 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 20:21:30.288678 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:21:30.288797 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":46,"thread_start_us":302,"threads_started":1}
I20260923 20:21:30.289920 15835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4261a94a734a2b932502c6ce126096" candidate_uuid: "68f2adbf738944788a693a70aba782e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2793939e389a4471b66a58c1cff3658e" is_pre_election: true
I20260923 20:21:30.290511 15835 raft_consensus.cc:2454] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f2adbf738944788a693a70aba782e8 in term 0.
I20260923 20:21:30.290555 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4261a94a734a2b932502c6ce126096" candidate_uuid: "68f2adbf738944788a693a70aba782e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a8a46156104231b6421be440cbb31d" is_pre_election: true
I20260923 20:21:30.291127 15702 raft_consensus.cc:2454] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68f2adbf738944788a693a70aba782e8 in term 0.
I20260923 20:21:30.291317 15501 leader_election.cc:302] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2793939e389a4471b66a58c1cff3658e, 68f2adbf738944788a693a70aba782e8; no voters:
I20260923 20:21:30.291716 15905 raft_consensus.cc:2781] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:30.291862 15905 raft_consensus.cc:482] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:30.291990 15905 raft_consensus.cc:3037] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:30.294279 15905 raft_consensus.cc:504] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.294907 15905 leader_election.cc:288] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [CANDIDATE]: Term 1 election: Requested vote from peers 2793939e389a4471b66a58c1cff3658e (127.14.127.195:43413), c9a8a46156104231b6421be440cbb31d (127.14.127.194:46601)
I20260923 20:21:30.295408 15835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4261a94a734a2b932502c6ce126096" candidate_uuid: "68f2adbf738944788a693a70aba782e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2793939e389a4471b66a58c1cff3658e"
I20260923 20:21:30.295418 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4261a94a734a2b932502c6ce126096" candidate_uuid: "68f2adbf738944788a693a70aba782e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a8a46156104231b6421be440cbb31d"
I20260923 20:21:30.295617 15702 raft_consensus.cc:3037] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:30.295601 15835 raft_consensus.cc:3037] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:30.297999 15702 raft_consensus.cc:2454] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f2adbf738944788a693a70aba782e8 in term 1.
I20260923 20:21:30.298192 15835 raft_consensus.cc:2454] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f2adbf738944788a693a70aba782e8 in term 1.
I20260923 20:21:30.298429 15501 leader_election.cc:302] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68f2adbf738944788a693a70aba782e8, c9a8a46156104231b6421be440cbb31d; no voters:
I20260923 20:21:30.298789 15905 raft_consensus.cc:2781] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:30.299305 15905 raft_consensus.cc:686] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [term 1 LEADER]: Becoming Leader. State: Replica: 68f2adbf738944788a693a70aba782e8, State: Running, Role: LEADER
I20260923 20:21:30.299836 15905 consensus_queue.cc:237] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } }
I20260923 20:21:30.304894 15424 catalog_manager.cc:5495] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68f2adbf738944788a693a70aba782e8 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "68f2adbf738944788a693a70aba782e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793939e389a4471b66a58c1cff3658e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a8a46156104231b6421be440cbb31d" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 46601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68f2adbf738944788a693a70aba782e8" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36665 } health_report { overall_health: HEALTHY } } }
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:21:30.345935 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.331305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.14.127.193:36665)
0923 20:21:30.331501 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.331505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.332004 (+ 499us) server_negotiation.cc:407] Connection header received
0923 20:21:30.332223 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.332227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.332307 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.332410 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:30.333895 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.334663 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.335915 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.336325 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.336451 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.340402 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:30.340416 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:30.340432 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:30.340464 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:30.343251 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.343897 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.343905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.343908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.344063 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.344428 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.344434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.344437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.344779 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:30.345069 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.345361 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.345623 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 20:21:30.346983 15881 tablet.cc:2369] T 6a4261a94a734a2b932502c6ce126096 P 2793939e389a4471b66a58c1cff3658e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260923 20:21:30.379837 15748 tablet.cc:2369] T 6a4261a94a734a2b932502c6ce126096 P c9a8a46156104231b6421be440cbb31d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194890, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
W20260923 20:21:30.386487 15614 tablet.cc:2369] T 6a4261a94a734a2b932502c6ce126096 P 68f2adbf738944788a693a70aba782e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:30.390228 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.376463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.14.127.193:36665)
0923 20:21:30.376697 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.376707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.377022 (+ 315us) server_negotiation.cc:407] Connection header received
0923 20:21:30.377240 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.377245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.377307 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.377412 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:30.378767 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.379492 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.380823 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.381228 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.381342 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.384910 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:30.384915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:30.384918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:30.384950 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:30.388097 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.388669 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.388672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.388675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.388794 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.389095 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.389098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.389100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.389341 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:30.389551 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.389720 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.389874 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:21:30.432866 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.420138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.14.127.193:36665)
0923 20:21:30.420369 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.420374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.420457 (+ 83us) server_negotiation.cc:407] Connection header received
0923 20:21:30.420608 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.420612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.420692 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.420778 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:30.422050 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.422881 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.424014 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.424476 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.424569 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.428335 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:30.428340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:30.428343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:30.428374 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:30.430728 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.431270 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.431274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.431276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.431393 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.431703 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.431706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.431708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.431945 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:30.432133 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.432336 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.432492 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260923 20:21:30.434808 15528 tablet_service.cc:2908] Rejecting scan request for tablet 6a4261a94a734a2b932502c6ce126096: Uninitialized: clean time has not yet been initialized
I20260923 20:21:30.438022 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 15485
I20260923 20:21:30.450002 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.438818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.14.127.193:36665)
0923 20:21:30.438965 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.438968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.439400 (+ 432us) server_negotiation.cc:407] Connection header received
0923 20:21:30.439491 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.439495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.439550 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.439622 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:30.440768 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.441563 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.442739 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.443143 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.443236 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.445175 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:30.445179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:30.445182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:30.445209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:30.447921 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.448517 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.448521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.448523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.448649 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.448965 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.448968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.448970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.449209 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:30.449391 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.449574 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.449743 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 20:21:30.450695 15608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:30.541599 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 15485
I20260923 20:21:30.558403 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 15619
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:21:30.574224 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.559327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.14.127.194:46601)
0923 20:21:30.559524 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.559529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.559550 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:21:30.559621 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.559626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.559702 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.559804 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:30.561032 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.562154 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.563512 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.564160 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.564279 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.568306 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:30.568326 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:30.568349 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:30.568393 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:30.571897 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.572504 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.572513 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.572517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.572681 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.572995 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.572998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.573000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.573354 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:30.573602 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.573817 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.573971 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:21:30.575052 15742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:30.663144 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 15619
I20260923 20:21:30.680152 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 15752
Sep 23 20:21:30 dist-test-slave-wmd0 krb5kdc[15379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:21:30.694388 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:30.680918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.14.127.195:43413)
0923 20:21:30.681092 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:30.681096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:30.681114 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:21:30.681242 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:30.681246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:30.681309 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:30.681395 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:30.682849 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.683633 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.684839 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:30.685226 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:30.685314 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:30.689524 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:30.689536 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:30.689552 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:30.689586 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:30.692196 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.692787 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.692794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.692796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.692918 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:30.693221 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:30.693224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:30.693225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:30.693545 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:30.693751 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:30.693982 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 20:21:30.694140 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:21:30.695062 15875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:30.783740 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 15752
I20260923 20:21:30.799970 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 15392
I20260923 20:21:30.801123 15454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:30.907513 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 15392
2026-09-23T20:21:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4036 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:31 dist-test-slave-wmd0 krb5kdc[15928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:31 dist-test-slave-wmd0 krb5kdc[15928](info): set up 2 sockets
Sep 23 20:21:31 dist-test-slave-wmd0 krb5kdc[15928](info): commencing operation
krb5kdc: starting...
W20260923 20:21:31.982766 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:21:32 dist-test-slave-wmd0 krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:32.165628 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36945
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:32.475505 15941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:32.475795 15941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:32.475965 15941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:32.483160 15941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:32.483264 15941 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:21:32.483366 15941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:32.483479 15941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:32.483568 15941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:32.483672 15941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:32.493685 15941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:36945
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:32.495021 15941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:32.496384 15941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:32.508108 15947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:32.508273 15949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:32.508648 15946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:32.509133 15941 server_base.cc:1034] running on GCE node
I20260923 20:21:32.509935 15941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:32.511426 15941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:32.512642 15941 hybrid_clock.cc:648] HybridClock initialized: now 1790194892512564 us; error 53 us; skew 500 ppm
I20260923 20:21:32.512986 15941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:32 dist-test-slave-wmd0 krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194892, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:32.516157 15941 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:32.517602 15941 webserver.cc:460] Webserver started at http://127.14.127.254:35009/ using document root <none> and password file <none>
I20260923 20:21:32.518335 15941 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:32.518469 15941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:32.518849 15941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:32.521862 15941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb"
format_stamp: "Formatted at 2026-09-23 20:21:32 on dist-test-slave-wmd0"
server_key: "00af569ec28f219f91c9f75ca3abe7e8"
server_key_iv: "eacf5a8f59b0ee3bfd00034c087ba722"
server_key_version: "encryptionkey@0"
I20260923 20:21:32.522697 15941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb"
format_stamp: "Formatted at 2026-09-23 20:21:32 on dist-test-slave-wmd0"
server_key: "00af569ec28f219f91c9f75ca3abe7e8"
server_key_iv: "eacf5a8f59b0ee3bfd00034c087ba722"
server_key_version: "encryptionkey@0"
I20260923 20:21:32.527789 15941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:21:32.531234 15956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:32.532594 15941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:21:32.532781 15941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb"
format_stamp: "Formatted at 2026-09-23 20:21:32 on dist-test-slave-wmd0"
server_key: "00af569ec28f219f91c9f75ca3abe7e8"
server_key_iv: "eacf5a8f59b0ee3bfd00034c087ba722"
server_key_version: "encryptionkey@0"
I20260923 20:21:32.533010 15941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:32.555508 15941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:32.560019 15941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:32.560308 15941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:32.582580 15941 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:36945
I20260923 20:21:32.582638 16017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:36945 every 8 connection(s)
I20260923 20:21:32.584033 15941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:32.588259 16018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:32.594972 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 15941
I20260923 20:21:32.595224 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:32.596019 14847 external_mini_cluster.cc:1428] Setting key 2a857cb4e8a50bb5bbe3dd768981cdc2
I20260923 20:21:32.601430 16018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb: Bootstrap starting.
I20260923 20:21:32.605190 16018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:32.606216 16018 log.cc:824] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:32.609328 16018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb: No bootstrap required, opened a new log
Sep 23 20:21:32 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:32.617919 16018 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36945 } }
I20260923 20:21:32.618331 16018 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:32.618464 16018 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b4fc6fcc2c4916a007ce2e09ddd3fb, State: Initialized, Role: FOLLOWER
I20260923 20:21:32.619212 16018 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36945 } }
I20260923 20:21:32.619500 16018 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:32.619642 16018 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:32.619820 16018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:32.620723 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:32.599570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.14.127.254:36945)
0923 20:21:32.600409 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:32.600421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:32.600463 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:21:32.602218 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:32.602239 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:32.602670 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:32.603028 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:32.604761 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:32.606159 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:32.607708 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:32.608338 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:32.608474 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:32.613114 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:32.613122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:32.613140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:32.613184 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:32.616239 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:32.616955 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:32.616971 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:32.616978 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:32.617154 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:32.617520 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:32.617525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:32.617528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:32.618058 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:32.618367 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:32.618823 (+ 456us) server_negotiation.cc:299] Negotiation successful
0923 20:21:32.619279 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":390,"threads_started":1}
I20260923 20:21:32.622470 16018 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36945 } }
I20260923 20:21:32.623013 16018 leader_election.cc:302] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1b4fc6fcc2c4916a007ce2e09ddd3fb; no voters:
I20260923 20:21:32.623667 16018 leader_election.cc:288] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:32.623788 16023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:32.624464 16023 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [term 1 LEADER]: Becoming Leader. State: Replica: e1b4fc6fcc2c4916a007ce2e09ddd3fb, State: Running, Role: LEADER
I20260923 20:21:32.625036 16023 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36945 } }
I20260923 20:21:32.625850 16018 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:32.628059 16024 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36945 } } }
I20260923 20:21:32.628149 16025 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b4fc6fcc2c4916a007ce2e09ddd3fb. Latest consensus state: current_term: 1 leader_uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b4fc6fcc2c4916a007ce2e09ddd3fb" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36945 } } }
I20260923 20:21:32.628448 16024 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:32.628527 16025 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:32.631824 16029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:32.636790 16029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:32.645630 16029 catalog_manager.cc:1347] Generated new cluster ID: 8bb1064c299344c88aba2653e97a6a88
I20260923 20:21:32.645743 16029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:32.684031 16029 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:32.685369 16029 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:32.698359 16029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb: Generated new TSK 0
I20260923 20:21:32.699208 16029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:21:32.751156 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36945
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:33.068817 16044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:33.069137 16044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:33.069324 16044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:33.077504 16044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:33.077649 16044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:33.077796 16044 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:21:33.077927 16044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:21:33.087597 16044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:36945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:33.088864 16044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:33.090178 16044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:33.103248 16049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:33.103312 16052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:33.103190 16050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:33.104179 16044 server_base.cc:1034] running on GCE node
I20260923 20:21:33.104806 16044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:33.105696 16044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:33.106925 16044 hybrid_clock.cc:648] HybridClock initialized: now 1790194893106851 us; error 53 us; skew 500 ppm
I20260923 20:21:33.107208 16044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:33 dist-test-slave-wmd0 krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:33.110270 16044 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:21:33.111579 16044 webserver.cc:460] Webserver started at http://127.14.127.193:38653/ using document root <none> and password file <none>
I20260923 20:21:33.112208 16044 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:33.112291 16044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:33.112550 16044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:33.115307 16044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "eeb6fa0551004e899022ab6923751761"
format_stamp: "Formatted at 2026-09-23 20:21:33 on dist-test-slave-wmd0"
server_key: "44e19d86938348f7d9c34cd694f6111e"
server_key_iv: "2efe050c8ec80bae591416e56f378bf3"
server_key_version: "encryptionkey@0"
I20260923 20:21:33.116041 16044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "eeb6fa0551004e899022ab6923751761"
format_stamp: "Formatted at 2026-09-23 20:21:33 on dist-test-slave-wmd0"
server_key: "44e19d86938348f7d9c34cd694f6111e"
server_key_iv: "2efe050c8ec80bae591416e56f378bf3"
server_key_version: "encryptionkey@0"
I20260923 20:21:33.121017 16044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:21:33.124351 16059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:33.125643 16044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:21:33.125808 16044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "eeb6fa0551004e899022ab6923751761"
format_stamp: "Formatted at 2026-09-23 20:21:33 on dist-test-slave-wmd0"
server_key: "44e19d86938348f7d9c34cd694f6111e"
server_key_iv: "2efe050c8ec80bae591416e56f378bf3"
server_key_version: "encryptionkey@0"
I20260923 20:21:33.126021 16044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:33.145186 16044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:33.149151 16044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:33.149428 16044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:33.152088 16044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:33.152170 16044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:33.152323 16044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:33.152410 16044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:21:33 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:33.174820 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:33.152543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:47967 (local address 127.14.127.254:36945)
0923 20:21:33.153357 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:33.153365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:33.154359 (+ 994us) server_negotiation.cc:407] Connection header received
0923 20:21:33.156351 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:33.156360 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:33.156469 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:33.156609 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:33.158799 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.159879 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.162497 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.163175 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.163307 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:33.167777 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:33.167785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:33.167792 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:33.167836 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:33.171049 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:33.172163 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:33.172172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:33.172179 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:33.172351 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:33.172949 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:33.172957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:33.172963 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:33.173339 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:33.173508 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:33.174191 (+ 683us) server_negotiation.cc:299] Negotiation successful
0923 20:21:33.174408 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":565,"threads_started":1}
I20260923 20:21:33.175930 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:33.153282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36945 (local address 127.14.127.193:47967)
0923 20:21:33.154157 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:33.154204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:33.156086 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:33.156884 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:33.156898 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:33.157621 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:33.158537 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:33.158566 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.160110 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.160117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:33.162341 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:33.162352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.163430 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.163441 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:33.163660 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:33.164471 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:33.164499 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:33.167510 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:33.171191 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:33.171205 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:33.171223 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:33.171997 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:33.172497 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:33.172501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:33.172506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:33.172759 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:33.173473 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:33.173481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:33.173834 (+ 353us) client_negotiation.cc:769] Sending connection context
0923 20:21:33.174102 (+ 268us) client_negotiation.cc:240] Negotiation successful
0923 20:21:33.174652 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":393,"threads_started":1}
I20260923 20:21:33.203203 16044 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:41651
I20260923 20:21:33.203224 16179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:41651 every 8 connection(s)
I20260923 20:21:33.203948 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:33.182472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36945 (local address 127.14.127.193:34617)
0923 20:21:33.191207 (+ 8735us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:33.191237 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:33.191411 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:33.191756 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:33.191761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:33.192217 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:33.192493 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:33.192503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.193502 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.193509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:33.194525 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:33.194536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.195493 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.195501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:33.195676 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:33.196380 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:33.196397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:33.197117 (+ 720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:33.201448 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:33.201452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:33.201455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:33.201827 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:33.202274 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:33.202281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:33.202286 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:33.202455 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:33.203014 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:33.203019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:33.203226 (+ 207us) client_negotiation.cc:769] Sending connection context
0923 20:21:33.203381 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 20:21:33.203638 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8571,"thread_start_us":8378,"threads_started":1}
I20260923 20:21:33.204581 16044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:21:33.204627 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:33.182469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:34617 (local address 127.14.127.254:36945)
0923 20:21:33.182702 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:33.182710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:33.191336 (+ 8626us) server_negotiation.cc:407] Connection header received
0923 20:21:33.191543 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:33.191547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:33.191627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:33.191730 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:33.192641 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.193365 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.194695 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.195281 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.195437 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:33.197325 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:33.197330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:33.197333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:33.197381 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:33.201328 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:33.201960 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:33.201968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:33.201971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:33.202103 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:33.202593 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:33.202601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:33.202625 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:33.202890 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:33.203053 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:33.204248 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0923 20:21:33.204407 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:21:33.206739 16180 heartbeater.cc:347] Connected to a master server at 127.14.127.254:36945
I20260923 20:21:33.206985 16180 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:33.207683 16180 heartbeater.cc:511] Master 127.14.127.254:36945 requested a full tablet report, sending...
I20260923 20:21:33.209249 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 16044
I20260923 20:21:33.209472 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:21:33.209911 15973 ts_manager.cc:194] Registered new tserver with Master: eeb6fa0551004e899022ab6923751761 (127.14.127.193:41651)
I20260923 20:21:33.210285 14847 external_mini_cluster.cc:1428] Setting key 6ecbb7acb9a962ddf3e966fcbedc3b34
I20260923 20:21:33.211773 15973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:47967
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:21:33.250252 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36945
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:33.568153 16184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:33.568423 16184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:33.568594 16184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:33.575982 16184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:33.576087 16184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:33.576225 16184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:21:33.576355 16184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:21:33.589277 16184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:36945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:33.590780 16184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:33.592415 16184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:33.605947 16189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:33.607733 16190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:33.607784 16192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:33.608322 16184 server_base.cc:1034] running on GCE node
I20260923 20:21:33.608934 16184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:33.609828 16184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:33.611032 16184 hybrid_clock.cc:648] HybridClock initialized: now 1790194893610948 us; error 59 us; skew 500 ppm
I20260923 20:21:33.611323 16184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:33 dist-test-slave-wmd0 krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:33.614286 16184 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:21:33.615818 16184 webserver.cc:460] Webserver started at http://127.14.127.194:40927/ using document root <none> and password file <none>
I20260923 20:21:33.616528 16184 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:33.616650 16184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:33.616966 16184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:33.619849 16184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "57923a2b414944ae8939894ae135dbac"
format_stamp: "Formatted at 2026-09-23 20:21:33 on dist-test-slave-wmd0"
server_key: "b8a8c5d291dd8f4869c175956e24c8d5"
server_key_iv: "791f47d48d6536b898fd4d757663b366"
server_key_version: "encryptionkey@0"
I20260923 20:21:33.620637 16184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "57923a2b414944ae8939894ae135dbac"
format_stamp: "Formatted at 2026-09-23 20:21:33 on dist-test-slave-wmd0"
server_key: "b8a8c5d291dd8f4869c175956e24c8d5"
server_key_iv: "791f47d48d6536b898fd4d757663b366"
server_key_version: "encryptionkey@0"
I20260923 20:21:33.625439 16184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:21:33.629061 16199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:33.630513 16184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:21:33.630702 16184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "57923a2b414944ae8939894ae135dbac"
format_stamp: "Formatted at 2026-09-23 20:21:33 on dist-test-slave-wmd0"
server_key: "b8a8c5d291dd8f4869c175956e24c8d5"
server_key_iv: "791f47d48d6536b898fd4d757663b366"
server_key_version: "encryptionkey@0"
I20260923 20:21:33.630920 16184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:33.644475 16184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:33.648346 16184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:33.648641 16184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:33.651196 16184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:33.651304 16184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:33.651443 16184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:33.651521 16184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:21:33 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:33.676409 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:33.651917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:39581 (local address 127.14.127.254:36945)
0923 20:21:33.652114 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:33.652119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:33.653661 (+ 1542us) server_negotiation.cc:407] Connection header received
0923 20:21:33.655783 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:33.655789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:33.655919 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:33.656069 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:33.658305 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.659349 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.661808 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.662289 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.662434 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:33.668485 (+ 6051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:33.668490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:33.668494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:33.668530 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:33.671994 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:33.672946 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:33.672957 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:33.672973 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:33.673194 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:33.673681 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:33.673690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:33.673696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:33.674234 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:33.674462 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:33.675403 (+ 941us) server_negotiation.cc:299] Negotiation successful
0923 20:21:33.675725 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:21:33.677625 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:33.652618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36945 (local address 127.14.127.194:39581)
0923 20:21:33.653487 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:33.653530 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:33.655451 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:33.656395 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:33.656415 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:33.657216 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:33.658126 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:33.658146 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.659505 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.659510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:33.661664 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:33.661676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.662670 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.662682 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:33.663212 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:33.664215 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:33.664262 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:33.668184 (+ 3922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:33.672175 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:33.672185 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:33.672208 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:33.672668 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:33.673338 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:33.673342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:33.673344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:33.673555 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:33.674396 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:33.674402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:33.674846 (+ 444us) client_negotiation.cc:769] Sending connection context
0923 20:21:33.675243 (+ 397us) client_negotiation.cc:240] Negotiation successful
0923 20:21:33.675805 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":383,"threads_started":1}
I20260923 20:21:33.701419 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:33.684974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:57511 (local address 127.14.127.254:36945)
0923 20:21:33.685192 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:33.685200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:33.685880 (+ 680us) server_negotiation.cc:407] Connection header received
0923 20:21:33.686155 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:33.686161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:33.686230 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:33.686335 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:33.687847 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.689271 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.691985 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.692391 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.692544 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:33.695264 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:33.695269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:33.695272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:33.695300 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:33.698767 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:33.699459 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:33.699465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:33.699468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:33.699596 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:33.699956 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:33.699960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:33.699964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:33.700278 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:33.700430 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:33.700853 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 20:21:33.701075 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:21:33.701556 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:33.684930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36945 (local address 127.14.127.194:57511)
0923 20:21:33.685743 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:33.685773 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:33.685939 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:33.686366 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:33.686371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:33.687061 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:33.687594 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:33.687611 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.689487 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.689497 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:33.691827 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:33.691851 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:33.692871 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:33.692876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:33.692990 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:33.693764 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:33.693782 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:33.695015 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:33.698952 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:33.698958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:33.698962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:33.699337 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:33.699702 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:33.699707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:33.699711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:33.699836 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:33.700394 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:33.700399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:33.700603 (+ 204us) client_negotiation.cc:769] Sending connection context
0923 20:21:33.700752 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 20:21:33.701071 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":414,"threads_started":1}
I20260923 20:21:33.707018 16184 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:44707
I20260923 20:21:33.707037 16318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:44707 every 8 connection(s)
I20260923 20:21:33.708415 16184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:21:33.710500 16319 heartbeater.cc:347] Connected to a master server at 127.14.127.254:36945
I20260923 20:21:33.710767 16319 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:33.711439 16319 heartbeater.cc:511] Master 127.14.127.254:36945 requested a full tablet report, sending...
I20260923 20:21:33.712740 15973 ts_manager.cc:194] Registered new tserver with Master: 57923a2b414944ae8939894ae135dbac (127.14.127.194:44707)
I20260923 20:21:33.713512 15973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:39581
I20260923 20:21:33.718286 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 16184
I20260923 20:21:33.718498 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:21:33.718997 14847 external_mini_cluster.cc:1428] Setting key 9282eff8bbf7a56243eb5fbf440ee2ff
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:21:33.758729 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36945
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:34.077298 16323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:34.077648 16323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:34.077803 16323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:34.085059 16323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:34.085203 16323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:34.085350 16323 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:21:34.085487 16323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:21:34.095057 16323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:36945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:34.096282 16323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:34.097599 16323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:34.110529 16329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:34.110559 16331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:34.111057 16328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:34.111621 16323 server_base.cc:1034] running on GCE node
I20260923 20:21:34.112187 16323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:34.113032 16323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:34.114272 16323 hybrid_clock.cc:648] HybridClock initialized: now 1790194894114197 us; error 56 us; skew 500 ppm
I20260923 20:21:34.114571 16323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:34 dist-test-slave-wmd0 krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:34.117604 16323 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:21:34.119091 16323 webserver.cc:460] Webserver started at http://127.14.127.195:45581/ using document root <none> and password file <none>
I20260923 20:21:34.119764 16323 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:34.119889 16323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:34.120196 16323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:34.123034 16323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "059ca99ca2a04940a7830c9b074cf00e"
format_stamp: "Formatted at 2026-09-23 20:21:34 on dist-test-slave-wmd0"
server_key: "d942ce264cbbec7f7cd17e23c4be6e97"
server_key_iv: "6f6d5f655580369ad5cb924a9a29c899"
server_key_version: "encryptionkey@0"
I20260923 20:21:34.123806 16323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "059ca99ca2a04940a7830c9b074cf00e"
format_stamp: "Formatted at 2026-09-23 20:21:34 on dist-test-slave-wmd0"
server_key: "d942ce264cbbec7f7cd17e23c4be6e97"
server_key_iv: "6f6d5f655580369ad5cb924a9a29c899"
server_key_version: "encryptionkey@0"
I20260923 20:21:34.128706 16323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:21:34.132192 16338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:34.133410 16323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:21:34.133584 16323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "059ca99ca2a04940a7830c9b074cf00e"
format_stamp: "Formatted at 2026-09-23 20:21:34 on dist-test-slave-wmd0"
server_key: "d942ce264cbbec7f7cd17e23c4be6e97"
server_key_iv: "6f6d5f655580369ad5cb924a9a29c899"
server_key_version: "encryptionkey@0"
I20260923 20:21:34.133793 16323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:34.149567 16323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:34.153283 16323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:34.153555 16323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:34.156126 16323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:34.156225 16323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:34.156365 16323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:34.156450 16323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:21:34 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194894, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:34.178843 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.156665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:37193 (local address 127.14.127.254:36945)
0923 20:21:34.156884 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.156892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.158185 (+ 1293us) server_negotiation.cc:407] Connection header received
0923 20:21:34.160329 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.160335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.160430 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.160540 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:34.162824 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.164121 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.166686 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.167332 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.167482 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.171221 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:34.171231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:34.171240 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:34.171294 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:34.175344 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:34.176320 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:34.176325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:34.176330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:34.176491 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:34.176915 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:34.176920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:34.176923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:34.177270 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:34.177448 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.178177 (+ 729us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.178416 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:21:34.179976 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.157270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36945 (local address 127.14.127.195:37193)
0923 20:21:34.158021 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.158070 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.159995 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.160821 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.160837 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.161702 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:34.162521 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.162542 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.164320 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.164327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.166518 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.166530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.167591 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.167598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.167797 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.168436 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:34.168457 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:34.170920 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:34.175543 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:34.175555 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:34.175566 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:34.176189 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:34.176607 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:34.176612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:34.176615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:34.176797 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:34.177425 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:34.177432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:34.177787 (+ 355us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.178038 (+ 251us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.178594 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":342,"threads_started":1}
I20260923 20:21:34.200098 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.185823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36945 (local address 127.14.127.195:37351)
0923 20:21:34.186767 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.186782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.186920 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.187239 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.187244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.187680 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:34.187948 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.187956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.188922 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.188928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.189896 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.189907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.190668 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.190675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.190877 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.191651 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:34.191677 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:34.192758 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:34.196861 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:34.196869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:34.196875 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:34.197553 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:34.197937 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:34.197945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:34.197951 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:34.198151 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:34.198644 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:34.198651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:34.198905 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.199110 (+ 205us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.199528 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":477,"threads_started":1}
I20260923 20:21:34.200546 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.185692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:37351 (local address 127.14.127.254:36945)
0923 20:21:34.185844 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.185848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.186859 (+ 1011us) server_negotiation.cc:407] Connection header received
0923 20:21:34.187041 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.187046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.187114 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.187215 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:34.188087 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.188786 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.190040 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.190453 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.190590 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.192974 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:34.192982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:34.192989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:34.193035 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:34.196711 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:34.197684 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:34.197690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:34.197693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:34.197823 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:34.198269 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:34.198272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:34.198274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:34.198514 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:34.198681 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.199936 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.200179 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 20:21:34.211581 16323 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:41467
I20260923 20:21:34.211601 16457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:41467 every 8 connection(s)
I20260923 20:21:34.212764 16323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:21:34.214334 16180 heartbeater.cc:503] Master 127.14.127.254:36945 was elected leader, sending a full tablet report...
I20260923 20:21:34.214913 16458 heartbeater.cc:347] Connected to a master server at 127.14.127.254:36945
I20260923 20:21:34.215158 16458 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:34.215855 16458 heartbeater.cc:511] Master 127.14.127.254:36945 requested a full tablet report, sending...
I20260923 20:21:34.217046 15973 ts_manager.cc:194] Registered new tserver with Master: 059ca99ca2a04940a7830c9b074cf00e (127.14.127.195:41467)
I20260923 20:21:34.217227 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 16323
I20260923 20:21:34.217412 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:21:34.217864 14847 external_mini_cluster.cc:1428] Setting key f368e40c6691c65556fb5409ee9444bd
I20260923 20:21:34.217891 15973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:37193
I20260923 20:21:34.220726 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:21:34 dist-test-slave-wmd0 krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:21:34 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:34.263572 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.250218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.14.127.254:36945)
0923 20:21:34.250389 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.250394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.250726 (+ 332us) server_negotiation.cc:407] Connection header received
0923 20:21:34.250892 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.250896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.250961 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.251062 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:34.252336 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.253039 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.254227 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.254680 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.254770 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.258387 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:34.258392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:34.258395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:34.258426 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:34.260812 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:34.261643 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:34.261651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:34.261656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:34.261838 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:34.262278 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:34.262283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:34.262287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:34.262568 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:34.262831 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.263041 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.263206 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:21:34.267473 15973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:21:34.274194 15973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:34.301873 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.286892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.14.127.193:41651)
0923 20:21:34.292983 (+ 6091us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.292996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.293031 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:21:34.293167 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.293179 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.293501 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.293737 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:34.294529 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.295963 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.299023 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.300523 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.300698 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.300871 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.301405 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.301578 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5888,"thread_start_us":5762,"threads_started":1}
I20260923 20:21:34.301967 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.286449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:41651 (local address 127.0.0.1:58580)
0923 20:21:34.292314 (+ 5865us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.292334 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.292440 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.293805 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.293816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.293861 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:34.294294 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.294307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.296184 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.296189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.298814 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.298830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.300935 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.300943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.301150 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.301205 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.301342 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.301528 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5726,"mutex_wait_us":6,"thread_start_us":581,"threads_started":1}
I20260923 20:21:34.302807 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.286286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:41467 (local address 127.0.0.1:44550)
0923 20:21:34.287329 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.287391 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.287592 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.296045 (+ 8453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.296052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.296083 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:34.296691 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.296714 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.298137 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.298144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.300078 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.300090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.301931 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.301941 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.302129 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.302164 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.302234 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.302373 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":2,"thread_start_us":5736,"threads_started":1}
I20260923 20:21:34.303332 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.286574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.14.127.195:41467)
0923 20:21:34.295481 (+ 8907us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.295487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.295512 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:21:34.295624 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.295632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.295846 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.296041 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:34.296919 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.298004 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.300217 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.301699 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.302567 (+ 868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.302764 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.302839 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.303013 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8722,"thread_start_us":7534,"threads_started":1}
I20260923 20:21:34.306022 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 270e974e4f9942deb4bbfbb62f839038 (DEFAULT_TABLE table=test-table [id=cd5847bc6b7647c694b57ae161b08fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:34.307096 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.292663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.14.127.194:44707)
0923 20:21:34.294765 (+ 2102us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.294773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.295409 (+ 636us) server_negotiation.cc:407] Connection header received
0923 20:21:34.295677 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.295684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.295846 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.296041 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:34.297029 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.298461 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.301031 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.302240 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.302425 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.302584 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.303154 (+ 570us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.306832 (+ 3678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1921,"thread_start_us":1782,"threads_started":1}
I20260923 20:21:34.306775 16388 tablet_service.cc:1467] Processing CreateTablet for tablet 270e974e4f9942deb4bbfbb62f839038 (DEFAULT_TABLE table=test-table [id=cd5847bc6b7647c694b57ae161b08fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:34.307307 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e974e4f9942deb4bbfbb62f839038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.307830 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.294759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44707 (local address 127.0.0.1:46686)
0923 20:21:34.295308 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.295325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.295503 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.296332 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.296335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.296355 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:34.296827 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.296843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.298674 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.298684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.300872 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.300887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.302753 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.302759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.302922 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.302961 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.307350 (+ 4389us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.307509 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":291,"threads_started":1}
I20260923 20:21:34.308173 16388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e974e4f9942deb4bbfbb62f839038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.311118 16249 tablet_service.cc:1467] Processing CreateTablet for tablet 270e974e4f9942deb4bbfbb62f839038 (DEFAULT_TABLE table=test-table [id=cd5847bc6b7647c694b57ae161b08fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:34.312399 16249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270e974e4f9942deb4bbfbb62f839038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.319500 16477 tablet_bootstrap.cc:492] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Bootstrap starting.
I20260923 20:21:34.321471 16478 tablet_bootstrap.cc:492] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Bootstrap starting.
I20260923 20:21:34.323185 16477 tablet_bootstrap.cc:654] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.324446 16477 log.cc:824] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:34.324887 16478 tablet_bootstrap.cc:654] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.325730 16480 tablet_bootstrap.cc:492] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Bootstrap starting.
I20260923 20:21:34.326516 16478 log.cc:824] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:34.329811 16477 tablet_bootstrap.cc:492] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: No bootstrap required, opened a new log
I20260923 20:21:34.330264 16477 ts_tablet_manager.cc:1397] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 20:21:34.330602 16480 tablet_bootstrap.cc:654] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.331605 16480 log.cc:824] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:34.335263 16480 tablet_bootstrap.cc:492] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: No bootstrap required, opened a new log
I20260923 20:21:34.335333 16478 tablet_bootstrap.cc:492] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: No bootstrap required, opened a new log
I20260923 20:21:34.335529 16480 ts_tablet_manager.cc:1397] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:21:34.335590 16478 ts_tablet_manager.cc:1397] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260923 20:21:34.341748 16477 raft_consensus.cc:348] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
I20260923 20:21:34.342175 16477 raft_consensus.cc:374] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.342272 16477 raft_consensus.cc:729] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059ca99ca2a04940a7830c9b074cf00e, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.342893 16477 consensus_queue.cc:260] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
I20260923 20:21:34.344151 16458 heartbeater.cc:503] Master 127.14.127.254:36945 was elected leader, sending a full tablet report...
I20260923 20:21:34.344821 16477 ts_tablet_manager.cc:1428] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260923 20:21:34.343634 16480 raft_consensus.cc:348] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
I20260923 20:21:34.345280 16480 raft_consensus.cc:374] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.345400 16480 raft_consensus.cc:729] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57923a2b414944ae8939894ae135dbac, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.345973 16480 consensus_queue.cc:260] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
I20260923 20:21:34.345954 16478 raft_consensus.cc:348] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
I20260923 20:21:34.346359 16478 raft_consensus.cc:374] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.346455 16478 raft_consensus.cc:729] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb6fa0551004e899022ab6923751761, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.347205 16319 heartbeater.cc:503] Master 127.14.127.254:36945 was elected leader, sending a full tablet report...
I20260923 20:21:34.347226 16478 consensus_queue.cc:260] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
I20260923 20:21:34.348981 16478 ts_tablet_manager.cc:1428] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260923 20:21:34.349254 16480 ts_tablet_manager.cc:1428] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Time spent starting tablet: real 0.014s user 0.009s sys 0.001s
I20260923 20:21:34.448519 16483 raft_consensus.cc:482] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:34.448863 16483 raft_consensus.cc:504] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
I20260923 20:21:34.450655 16483 leader_election.cc:288] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb6fa0551004e899022ab6923751761 (127.14.127.193:41651), 57923a2b414944ae8939894ae135dbac (127.14.127.194:44707)
W20260923 20:21:34.457274 16181 tablet.cc:2369] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:34.458030 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.451048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:41651 (local address 127.14.127.195:55191)
0923 20:21:34.451346 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.451369 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.451536 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.451936 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.451942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.452025 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:34.452244 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.452256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.453516 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.453519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.455655 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.455670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.457206 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.457212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.457406 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.457446 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.457547 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.457694 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":87,"spinlock_wait_cycles":24320}
I20260923 20:21:34.458254 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.451093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:43597 (local address 127.14.127.194:44707)
0923 20:21:34.451314 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.451323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.451957 (+ 634us) server_negotiation.cc:407] Connection header received
0923 20:21:34.452114 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.452118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.452173 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.452242 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:34.452792 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.453740 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.456063 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.457279 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.457487 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.457559 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.457875 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.458014 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:21:34.458501 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.451117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:55191 (local address 127.14.127.193:41651)
0923 20:21:34.451313 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.451319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.451486 (+ 167us) server_negotiation.cc:407] Connection header received
0923 20:21:34.451707 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.451713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.451800 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.451880 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:34.452403 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.453388 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.455817 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.457039 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.457212 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.457290 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.457758 (+ 468us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.457903 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:21:34.458889 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.451101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44707 (local address 127.14.127.195:43597)
0923 20:21:34.451817 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.451830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.451975 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.452294 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.452301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.452331 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:34.452633 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.452641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.453876 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.453881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.455917 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.455926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.457507 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.457513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.457682 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.457722 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.458362 (+ 640us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.458493 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":25,"thread_start_us":380,"threads_started":1}
I20260923 20:21:34.459470 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e974e4f9942deb4bbfbb62f839038" candidate_uuid: "059ca99ca2a04940a7830c9b074cf00e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb6fa0551004e899022ab6923751761" is_pre_election: true
I20260923 20:21:34.459887 16273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e974e4f9942deb4bbfbb62f839038" candidate_uuid: "059ca99ca2a04940a7830c9b074cf00e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57923a2b414944ae8939894ae135dbac" is_pre_election: true
I20260923 20:21:34.460071 16134 raft_consensus.cc:2454] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059ca99ca2a04940a7830c9b074cf00e in term 0.
W20260923 20:21:34.460501 16320 tablet.cc:2369] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:34.460510 16273 raft_consensus.cc:2454] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059ca99ca2a04940a7830c9b074cf00e in term 0.
I20260923 20:21:34.460880 16339 leader_election.cc:302] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059ca99ca2a04940a7830c9b074cf00e, eeb6fa0551004e899022ab6923751761; no voters:
I20260923 20:21:34.461313 16483 raft_consensus.cc:2781] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:34.461464 16483 raft_consensus.cc:482] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:34.461594 16483 raft_consensus.cc:3037] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:34.463904 16483 raft_consensus.cc:504] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
W20260923 20:21:34.464206 16459 tablet.cc:2369] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:34.464632 16483 leader_election.cc:288] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [CANDIDATE]: Term 1 election: Requested vote from peers eeb6fa0551004e899022ab6923751761 (127.14.127.193:41651), 57923a2b414944ae8939894ae135dbac (127.14.127.194:44707)
I20260923 20:21:34.465092 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e974e4f9942deb4bbfbb62f839038" candidate_uuid: "059ca99ca2a04940a7830c9b074cf00e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb6fa0551004e899022ab6923751761"
I20260923 20:21:34.465152 16273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270e974e4f9942deb4bbfbb62f839038" candidate_uuid: "059ca99ca2a04940a7830c9b074cf00e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57923a2b414944ae8939894ae135dbac"
I20260923 20:21:34.465322 16134 raft_consensus.cc:3037] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:34.465332 16273 raft_consensus.cc:3037] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:34.468853 16134 raft_consensus.cc:2454] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059ca99ca2a04940a7830c9b074cf00e in term 1.
I20260923 20:21:34.469000 16273 raft_consensus.cc:2454] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059ca99ca2a04940a7830c9b074cf00e in term 1.
I20260923 20:21:34.469496 16341 leader_election.cc:302] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059ca99ca2a04940a7830c9b074cf00e, 57923a2b414944ae8939894ae135dbac; no voters:
I20260923 20:21:34.469871 16483 raft_consensus.cc:2781] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:34.470505 16483 raft_consensus.cc:686] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 1 LEADER]: Becoming Leader. State: Replica: 059ca99ca2a04940a7830c9b074cf00e, State: Running, Role: LEADER
I20260923 20:21:34.470986 16483 consensus_queue.cc:237] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } }
I20260923 20:21:34.476184 15972 catalog_manager.cc:5495] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e reported cstate change: term changed from 0 to 1, leader changed from <none> to 059ca99ca2a04940a7830c9b074cf00e (127.14.127.195). New cstate: current_term: 1 leader_uuid: "059ca99ca2a04940a7830c9b074cf00e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } health_report { overall_health: UNKNOWN } } }
I20260923 20:21:34.550153 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.544399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.14.127.195:41467)
0923 20:21:34.544579 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.544583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.544850 (+ 267us) server_negotiation.cc:407] Connection header received
0923 20:21:34.545175 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.545178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.545255 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.545383 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:34.545731 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.546448 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.547920 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.548411 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.548503 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.549002 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:34.549512 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:34.549653 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.549870 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.549940 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:21:34.559324 16134 raft_consensus.cc:1261] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Refusing update from remote peer 059ca99ca2a04940a7830c9b074cf00e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:34.560425 16487 consensus_queue.cc:1035] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:34.561496 16273 raft_consensus.cc:1261] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Refusing update from remote peer 059ca99ca2a04940a7830c9b074cf00e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:34.562286 16487 consensus_queue.cc:1035] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:34.593421 15972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:34.593724 15972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:34.597473 15972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb: Sending DeleteTablet for 3 replicas of tablet 270e974e4f9942deb4bbfbb62f839038
I20260923 20:21:34.598798 16388 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e974e4f9942deb4bbfbb62f839038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:34 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:44550
I20260923 20:21:34.598809 16249 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e974e4f9942deb4bbfbb62f839038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:34 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:46686
I20260923 20:21:34.599542 16110 tablet_service.cc:1514] Processing DeleteTablet for tablet 270e974e4f9942deb4bbfbb62f839038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:34 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:58580
I20260923 20:21:34.600808 15972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:21:34.601564 15972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:34.612689 16248 tablet_service.cc:1467] Processing CreateTablet for tablet 38660c54a1444fd9a2900aa521830b98 (DEFAULT_TABLE table=test-table [id=e57bc47bcf3a4c76a8d12d3cc7ae6f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:34.613577 16248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38660c54a1444fd9a2900aa521830b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.615370 16502 tablet_replica.cc:330] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: stopping tablet replica
I20260923 20:21:34.615578 16387 tablet_service.cc:1467] Processing CreateTablet for tablet 38660c54a1444fd9a2900aa521830b98 (DEFAULT_TABLE table=test-table [id=e57bc47bcf3a4c76a8d12d3cc7ae6f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:34.615876 16502 raft_consensus.cc:2229] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:34.616246 16387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38660c54a1444fd9a2900aa521830b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.616425 16502 raft_consensus.cc:2258] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:34.617259 16501 tablet_replica.cc:330] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: stopping tablet replica
I20260923 20:21:34.617676 16501 raft_consensus.cc:2229] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:34.618191 16501 raft_consensus.cc:2258] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:34.620501 16500 tablet_replica.cc:330] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: stopping tablet replica
I20260923 20:21:34.620934 16500 raft_consensus.cc:2229] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:21:34.621785 16500 raft_consensus.cc:2258] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:34.623154 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 38660c54a1444fd9a2900aa521830b98 (DEFAULT_TABLE table=test-table [id=e57bc47bcf3a4c76a8d12d3cc7ae6f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:34.623530 16477 tablet_bootstrap.cc:492] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: Bootstrap starting.
I20260923 20:21:34.624462 16501 ts_tablet_manager.cc:1905] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:34.625551 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38660c54a1444fd9a2900aa521830b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.626291 16477 tablet_bootstrap.cc:654] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.627161 16502 ts_tablet_manager.cc:1905] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:34.630343 16480 tablet_bootstrap.cc:492] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: Bootstrap starting.
I20260923 20:21:34.633287 16480 tablet_bootstrap.cc:654] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.635528 16502 ts_tablet_manager.cc:1918] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:21:34.635730 16502 log.cc:1196] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/270e974e4f9942deb4bbfbb62f839038
I20260923 20:21:34.636432 16502 ts_tablet_manager.cc:1939] T 270e974e4f9942deb4bbfbb62f839038 P eeb6fa0551004e899022ab6923751761: Deleting consensus metadata
I20260923 20:21:34.636642 16500 ts_tablet_manager.cc:1905] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:34.637015 16478 tablet_bootstrap.cc:492] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: Bootstrap starting.
I20260923 20:21:34.638597 16478 tablet_bootstrap.cc:654] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.640271 16501 ts_tablet_manager.cc:1918] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:21:34.640522 16501 log.cc:1196] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/270e974e4f9942deb4bbfbb62f839038
I20260923 20:21:34.641186 16501 ts_tablet_manager.cc:1939] T 270e974e4f9942deb4bbfbb62f839038 P 57923a2b414944ae8939894ae135dbac: Deleting consensus metadata
I20260923 20:21:34.643532 15959 catalog_manager.cc:4841] TS 57923a2b414944ae8939894ae135dbac (127.14.127.194:44707): tablet 270e974e4f9942deb4bbfbb62f839038 (table test-table [id=cd5847bc6b7647c694b57ae161b08fc1]) successfully deleted
I20260923 20:21:34.644372 15957 catalog_manager.cc:4841] TS eeb6fa0551004e899022ab6923751761 (127.14.127.193:41651): tablet 270e974e4f9942deb4bbfbb62f839038 (table test-table [id=cd5847bc6b7647c694b57ae161b08fc1]) successfully deleted
I20260923 20:21:34.648875 16477 tablet_bootstrap.cc:492] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: No bootstrap required, opened a new log
I20260923 20:21:34.649111 16477 ts_tablet_manager.cc:1397] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.005s
I20260923 20:21:34.650280 16478 tablet_bootstrap.cc:492] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: No bootstrap required, opened a new log
I20260923 20:21:34.650447 16478 ts_tablet_manager.cc:1397] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20260923 20:21:34.651263 16478 raft_consensus.cc:348] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.651437 16480 tablet_bootstrap.cc:492] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: No bootstrap required, opened a new log
I20260923 20:21:34.651503 16478 raft_consensus.cc:374] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.651576 16480 ts_tablet_manager.cc:1397] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20260923 20:21:34.651605 16478 raft_consensus.cc:729] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb6fa0551004e899022ab6923751761, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.649993 16477 raft_consensus.cc:348] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.652405 16480 raft_consensus.cc:348] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.652617 16480 raft_consensus.cc:374] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.652710 16480 raft_consensus.cc:729] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57923a2b414944ae8939894ae135dbac, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.653003 16480 consensus_queue.cc:260] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.653723 16480 ts_tablet_manager.cc:1428] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 20:21:34.652071 16478 consensus_queue.cc:260] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.654157 16477 raft_consensus.cc:374] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.654497 16478 ts_tablet_manager.cc:1428] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260923 20:21:34.656245 16485 raft_consensus.cc:482] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:34.656575 16485 raft_consensus.cc:504] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.657403 16477 raft_consensus.cc:729] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059ca99ca2a04940a7830c9b074cf00e, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.657958 16477 consensus_queue.cc:260] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.658943 16485 leader_election.cc:288] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57923a2b414944ae8939894ae135dbac (127.14.127.194:44707), 059ca99ca2a04940a7830c9b074cf00e (127.14.127.195:41467)
I20260923 20:21:34.659152 16477 ts_tablet_manager.cc:1428] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20260923 20:21:34.663256 16500 ts_tablet_manager.cc:1918] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:21:34.663470 16500 log.cc:1196] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/270e974e4f9942deb4bbfbb62f839038
I20260923 20:21:34.664089 16500 ts_tablet_manager.cc:1939] T 270e974e4f9942deb4bbfbb62f839038 P 059ca99ca2a04940a7830c9b074cf00e: Deleting consensus metadata
I20260923 20:21:34.665772 15959 catalog_manager.cc:4841] TS 059ca99ca2a04940a7830c9b074cf00e (127.14.127.195:41467): tablet 270e974e4f9942deb4bbfbb62f839038 (table test-table [id=cd5847bc6b7647c694b57ae161b08fc1]) successfully deleted
I20260923 20:21:34.668808 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.659960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:50203 (local address 127.14.127.194:44707)
0923 20:21:34.660188 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.660194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.661141 (+ 947us) server_negotiation.cc:407] Connection header received
0923 20:21:34.661417 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.661425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.661501 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.661567 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:34.662590 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.663717 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.666762 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.667760 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.667954 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.668028 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.668309 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.668431 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 20:21:34.669669 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.659974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44707 (local address 127.14.127.193:50203)
0923 20:21:34.661029 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.661051 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.661246 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.661885 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.661889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.661928 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:34.662374 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.662387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.663889 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.663895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.666582 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.666600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.667976 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.667983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.668152 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.668176 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.668285 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.668426 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"thread_start_us":2075,"threads_started":1}
I20260923 20:21:34.670321 16273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38660c54a1444fd9a2900aa521830b98" candidate_uuid: "eeb6fa0551004e899022ab6923751761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57923a2b414944ae8939894ae135dbac" is_pre_election: true
I20260923 20:21:34.670591 16273 raft_consensus.cc:2454] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb6fa0551004e899022ab6923751761 in term 0.
I20260923 20:21:34.671121 16062 leader_election.cc:302] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57923a2b414944ae8939894ae135dbac, eeb6fa0551004e899022ab6923751761; no voters:
I20260923 20:21:34.671531 16485 raft_consensus.cc:2781] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:34.671679 16485 raft_consensus.cc:482] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:34.671783 16485 raft_consensus.cc:3037] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:34.674587 16485 raft_consensus.cc:504] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.675315 16485 leader_election.cc:288] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [CANDIDATE]: Term 1 election: Requested vote from peers 57923a2b414944ae8939894ae135dbac (127.14.127.194:44707), 059ca99ca2a04940a7830c9b074cf00e (127.14.127.195:41467)
I20260923 20:21:34.675882 16273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38660c54a1444fd9a2900aa521830b98" candidate_uuid: "eeb6fa0551004e899022ab6923751761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57923a2b414944ae8939894ae135dbac"
I20260923 20:21:34.676105 16273 raft_consensus.cc:3037] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:34.677021 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.662888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:50659 (local address 127.14.127.195:41467)
0923 20:21:34.663051 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.663057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.663175 (+ 118us) server_negotiation.cc:407] Connection header received
0923 20:21:34.663498 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.663505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.663590 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.663665 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:34.667449 (+ 3784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.668557 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.673816 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.675275 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.675442 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.675521 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.676500 (+ 979us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.676656 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:21:34.678108 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.662495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:41467 (local address 127.14.127.193:50659)
0923 20:21:34.663086 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.663109 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.663270 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.663950 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.663955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.663980 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:34.667225 (+ 3245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.667240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.668751 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.668756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.673594 (+ 4838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.673615 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.675986 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.675994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.676296 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.676333 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.677564 (+ 1231us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.677718 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":553,"threads_started":1}
I20260923 20:21:34.678400 16273 raft_consensus.cc:2454] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb6fa0551004e899022ab6923751761 in term 1.
I20260923 20:21:34.678922 16062 leader_election.cc:302] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57923a2b414944ae8939894ae135dbac, eeb6fa0551004e899022ab6923751761; no voters:
I20260923 20:21:34.679224 16485 raft_consensus.cc:2781] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:34.679049 16413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38660c54a1444fd9a2900aa521830b98" candidate_uuid: "eeb6fa0551004e899022ab6923751761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059ca99ca2a04940a7830c9b074cf00e" is_pre_election: true
I20260923 20:21:34.679548 16413 raft_consensus.cc:2454] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eeb6fa0551004e899022ab6923751761 in term 0.
I20260923 20:21:34.679774 16485 raft_consensus.cc:686] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 1 LEADER]: Becoming Leader. State: Replica: eeb6fa0551004e899022ab6923751761, State: Running, Role: LEADER
I20260923 20:21:34.680213 16485 consensus_queue.cc:237] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.680529 16413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38660c54a1444fd9a2900aa521830b98" candidate_uuid: "eeb6fa0551004e899022ab6923751761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059ca99ca2a04940a7830c9b074cf00e"
I20260923 20:21:34.680727 16413 raft_consensus.cc:3037] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:34.684072 16413 raft_consensus.cc:2454] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eeb6fa0551004e899022ab6923751761 in term 1.
I20260923 20:21:34.683976 15972 catalog_manager.cc:5495] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 reported cstate change: term changed from 0 to 1, leader changed from <none> to eeb6fa0551004e899022ab6923751761 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "eeb6fa0551004e899022ab6923751761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:21:34 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194892, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:34.739974 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.717788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.14.127.254:36945)
0923 20:21:34.717965 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:34.717972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:34.718647 (+ 675us) server_negotiation.cc:407] Connection header received
0923 20:21:34.718829 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:34.718834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:34.718913 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:34.719024 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:34.720451 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.721628 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.723483 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.724206 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.724342 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.733970 (+ 9628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:34.733975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:34.733978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:34.734007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:34.736878 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:34.737874 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:34.737878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:34.737880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:34.738006 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:34.738405 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:34.738408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:34.738410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:34.738750 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:34.738876 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:34.739366 (+ 490us) server_negotiation.cc:299] Negotiation successful
0923 20:21:34.739709 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":61}
I20260923 20:21:34.740064 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:34.717862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36945 (local address 127.0.0.1:33322)
0923 20:21:34.718501 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:34.718524 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:34.718701 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:34.719062 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:34.719067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:34.719931 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:34.720295 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.720306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.721770 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.721775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.723330 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:34.723348 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:34.724395 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:34.724401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:34.724569 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:34.731258 (+ 6689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:34.731276 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:34.733761 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:34.737004 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:34.737017 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:34.737025 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:34.737710 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:34.738113 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:34.738120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:34.738124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:34.738300 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:34.738874 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:34.738883 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:34.739099 (+ 216us) client_negotiation.cc:769] Sending connection context
0923 20:21:34.739275 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 20:21:34.739582 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":353,"threads_started":1}
I20260923 20:21:34.745559 15972 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:33322:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:21:34.746920 15972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:34.753813 16110 tablet_service.cc:1467] Processing CreateTablet for tablet 49eba7af38064adaab574b88e5449659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a39e29e579064652875a240745d076ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:21:34.754070 16248 tablet_service.cc:1467] Processing CreateTablet for tablet 49eba7af38064adaab574b88e5449659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a39e29e579064652875a240745d076ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:21:34.754412 16110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eba7af38064adaab574b88e5449659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.754433 16387 tablet_service.cc:1467] Processing CreateTablet for tablet 49eba7af38064adaab574b88e5449659 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a39e29e579064652875a240745d076ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:21:34.754654 16248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eba7af38064adaab574b88e5449659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.754944 16387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eba7af38064adaab574b88e5449659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:34.761950 16478 tablet_bootstrap.cc:492] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: Bootstrap starting.
I20260923 20:21:34.762748 16477 tablet_bootstrap.cc:492] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: Bootstrap starting.
I20260923 20:21:34.763705 16478 tablet_bootstrap.cc:654] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.764067 16477 tablet_bootstrap.cc:654] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.764758 16480 tablet_bootstrap.cc:492] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: Bootstrap starting.
I20260923 20:21:34.766399 16480 tablet_bootstrap.cc:654] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:34.768203 16478 tablet_bootstrap.cc:492] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: No bootstrap required, opened a new log
I20260923 20:21:34.768394 16478 ts_tablet_manager.cc:1397] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260923 20:21:34.769366 16478 raft_consensus.cc:348] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.769673 16478 raft_consensus.cc:374] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.769757 16478 raft_consensus.cc:729] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb6fa0551004e899022ab6923751761, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.770056 16478 consensus_queue.cc:260] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.772122 16485 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:34.772522 16485 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:34.773079 16478 ts_tablet_manager.cc:1428] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260923 20:21:34.777017 16477 tablet_bootstrap.cc:492] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: No bootstrap required, opened a new log
I20260923 20:21:34.777220 16477 ts_tablet_manager.cc:1397] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.002s
I20260923 20:21:34.778003 16477 raft_consensus.cc:348] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.778234 16477 raft_consensus.cc:374] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.778318 16477 raft_consensus.cc:729] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059ca99ca2a04940a7830c9b074cf00e, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.778704 16477 consensus_queue.cc:260] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.779362 16495 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:34.779683 16495 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:34.780247 16480 tablet_bootstrap.cc:492] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: No bootstrap required, opened a new log
I20260923 20:21:34.780606 16480 ts_tablet_manager.cc:1397] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20260923 20:21:34.780689 16477 ts_tablet_manager.cc:1428] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:21:34.781834 16480 raft_consensus.cc:348] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.782222 16480 raft_consensus.cc:374] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:34.782387 16480 raft_consensus.cc:729] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57923a2b414944ae8939894ae135dbac, State: Initialized, Role: FOLLOWER
I20260923 20:21:34.782797 16480 consensus_queue.cc:260] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:34.783460 16484 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:34.784055 16484 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:34.784969 16480 ts_tablet_manager.cc:1428] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:21:35.091879 16484 raft_consensus.cc:482] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:35.092123 16484 raft_consensus.cc:504] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:35.093628 16484 leader_election.cc:288] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeb6fa0551004e899022ab6923751761 (127.14.127.193:41651), 059ca99ca2a04940a7830c9b074cf00e (127.14.127.195:41467)
I20260923 20:21:35.104489 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.094427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:36087 (local address 127.14.127.195:41467)
0923 20:21:35.094593 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.094599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.098633 (+ 4034us) server_negotiation.cc:407] Connection header received
0923 20:21:35.098841 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.098845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.098911 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.099011 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:35.099780 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.100513 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.102587 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.103374 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.103510 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.103563 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.104048 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.104209 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:21:35.104795 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.094179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:44515 (local address 127.14.127.193:41651)
0923 20:21:35.098694 (+ 4515us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.098700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.098725 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:21:35.098787 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.098791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.098846 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.098965 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:35.099793 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.100560 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.102686 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.103717 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.103955 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.104041 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.104366 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.104508 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4363,"thread_start_us":4262,"threads_started":1}
I20260923 20:21:35.105363 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.093924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:41651 (local address 127.14.127.194:44515)
0923 20:21:35.098100 (+ 4176us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:35.098129 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:35.098306 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:35.098984 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:35.098988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:35.099022 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:35.099433 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.099445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.100713 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.100720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.102434 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.102446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.104016 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.104023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.104186 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.104221 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:21:35.104342 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 20:21:35.104472 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4017,"thread_start_us":3858,"threads_started":1}
I20260923 20:21:35.105535 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.094046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:41467 (local address 127.14.127.194:36087)
0923 20:21:35.098483 (+ 4437us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:35.098501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:35.098695 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:35.099024 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:35.099029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:35.099052 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:35.099481 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.099490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.100652 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.100657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.102411 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.102423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.103623 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.103630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.103825 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.103866 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:21:35.104809 (+ 943us) client_negotiation.cc:240] Negotiation successful
0923 20:21:35.104947 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4288,"mutex_wait_us":74,"thread_start_us":4075,"threads_started":1}
I20260923 20:21:35.106112 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eba7af38064adaab574b88e5449659" candidate_uuid: "57923a2b414944ae8939894ae135dbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb6fa0551004e899022ab6923751761" is_pre_election: true
I20260923 20:21:35.106057 16485 consensus_queue.cc:1035] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:35.106357 16134 raft_consensus.cc:2454] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57923a2b414944ae8939894ae135dbac in term 0.
I20260923 20:21:35.106328 16413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eba7af38064adaab574b88e5449659" candidate_uuid: "57923a2b414944ae8939894ae135dbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059ca99ca2a04940a7830c9b074cf00e" is_pre_election: true
I20260923 20:21:35.106758 16413 raft_consensus.cc:2454] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57923a2b414944ae8939894ae135dbac in term 0.
I20260923 20:21:35.106837 16200 leader_election.cc:302] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57923a2b414944ae8939894ae135dbac, eeb6fa0551004e899022ab6923751761; no voters:
I20260923 20:21:35.107362 16484 raft_consensus.cc:2781] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:35.107581 16484 raft_consensus.cc:482] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:35.107769 16484 raft_consensus.cc:3037] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:35.111245 16484 raft_consensus.cc:504] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:35.112911 16134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eba7af38064adaab574b88e5449659" candidate_uuid: "57923a2b414944ae8939894ae135dbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeb6fa0551004e899022ab6923751761"
I20260923 20:21:35.113238 16134 raft_consensus.cc:3037] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:35.114039 16484 leader_election.cc:288] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [CANDIDATE]: Term 1 election: Requested vote from peers eeb6fa0551004e899022ab6923751761 (127.14.127.193:41651), 059ca99ca2a04940a7830c9b074cf00e (127.14.127.195:41467)
I20260923 20:21:35.117019 16134 raft_consensus.cc:2454] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57923a2b414944ae8939894ae135dbac in term 1.
I20260923 20:21:35.117233 16485 consensus_queue.cc:1035] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:35.118271 16200 leader_election.cc:302] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57923a2b414944ae8939894ae135dbac, eeb6fa0551004e899022ab6923751761; no voters:
I20260923 20:21:35.118381 16412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eba7af38064adaab574b88e5449659" candidate_uuid: "57923a2b414944ae8939894ae135dbac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059ca99ca2a04940a7830c9b074cf00e"
I20260923 20:21:35.118588 16412 raft_consensus.cc:3037] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:35.118956 16484 raft_consensus.cc:2781] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:35.119215 16484 raft_consensus.cc:686] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [term 1 LEADER]: Becoming Leader. State: Replica: 57923a2b414944ae8939894ae135dbac, State: Running, Role: LEADER
I20260923 20:21:35.119843 16484 consensus_queue.cc:237] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } }
I20260923 20:21:35.121563 16412 raft_consensus.cc:2454] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57923a2b414944ae8939894ae135dbac in term 1.
I20260923 20:21:35.122424 16522 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: TxnStatusTablet state changed. Reason: New leader 57923a2b414944ae8939894ae135dbac. Latest consensus state: current_term: 1 leader_uuid: "57923a2b414944ae8939894ae135dbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:35.122887 16522 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:21:35.124358 16529 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:21:35.124626 16529 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:21:35.125078 15972 catalog_manager.cc:5495] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac reported cstate change: term changed from 0 to 1, leader changed from <none> to 57923a2b414944ae8939894ae135dbac (127.14.127.194). New cstate: current_term: 1 leader_uuid: "57923a2b414944ae8939894ae135dbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } health_report { overall_health: UNKNOWN } } }
I20260923 20:21:35.144930 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.138790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44707 (local address 127.0.0.1:46688)
0923 20:21:35.139014 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:35.139033 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:35.139181 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:35.139686 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:35.139689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:35.139699 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:21:35.139844 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.139851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.141255 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.141258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.142257 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.142268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.143279 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.143284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.143440 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.143494 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:21:35.144462 (+ 968us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:21:35.144482 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 20:21:35.144555 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 20:21:35.144681 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 20:21:35.144977 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.139099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.14.127.194:44707)
0923 20:21:35.139302 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.139308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.139329 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:21:35.139412 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.139417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.139507 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.139658 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:35.139968 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.141142 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.142397 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.143093 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.143239 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.143696 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:35.144334 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:35.144480 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.144609 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.144693 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:21:35.149637 16413 raft_consensus.cc:1261] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e [term 1 FOLLOWER]: Refusing update from remote peer 57923a2b414944ae8939894ae135dbac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:35.149645 16134 raft_consensus.cc:1261] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Refusing update from remote peer 57923a2b414944ae8939894ae135dbac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:35.150301 16522 consensus_queue.cc:1035] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:35.150666 16484 consensus_queue.cc:1035] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:35.153673 16495 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: TxnStatusTablet state changed. Reason: New leader 57923a2b414944ae8939894ae135dbac. Latest consensus state: current_term: 1 leader_uuid: "57923a2b414944ae8939894ae135dbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:35.153959 16495 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:35.156203 16525 mvcc.cc:205] Tried to move back new op lower bound from 7332638290526146560 to 7332638290416328704. Current Snapshot: MvccSnapshot[applied={T|T < 7332638290526146560}]
I20260923 20:21:35.157424 16527 mvcc.cc:205] Tried to move back new op lower bound from 7332638290526146560 to 7332638290416328704. Current Snapshot: MvccSnapshot[applied={T|T < 7332638290526146560}]
I20260923 20:21:35.163020 16522 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57923a2b414944ae8939894ae135dbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:35.163300 16522 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:21:35.164930 16495 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57923a2b414944ae8939894ae135dbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:35.165170 16495 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P 059ca99ca2a04940a7830c9b074cf00e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:35.167506 16508 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: TxnStatusTablet state changed. Reason: New leader 57923a2b414944ae8939894ae135dbac. Latest consensus state: current_term: 1 leader_uuid: "57923a2b414944ae8939894ae135dbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:35.167902 16508 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:35.169612 16484 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57923a2b414944ae8939894ae135dbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:35.169889 16484 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P 57923a2b414944ae8939894ae135dbac: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:21:35.176271 16508 tablet_replica.cc:439] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57923a2b414944ae8939894ae135dbac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeb6fa0551004e899022ab6923751761" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 41651 } } peers { permanent_uuid: "57923a2b414944ae8939894ae135dbac" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44707 } } peers { permanent_uuid: "059ca99ca2a04940a7830c9b074cf00e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41467 } } }
I20260923 20:21:35.176517 16508 tablet_replica.cc:442] T 49eba7af38064adaab574b88e5449659 P eeb6fa0551004e899022ab6923751761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:35.180034 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.171569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.14.127.193:41651)
0923 20:21:35.171738 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.171746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.173575 (+ 1829us) server_negotiation.cc:407] Connection header received
0923 20:21:35.173812 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.173816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.173910 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.174070 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:35.174546 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.175683 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.177747 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.178185 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.178320 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.178806 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:35.179481 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:35.179649 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.179699 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.179775 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 20:21:35.195169 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.189688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:46093 (local address 127.14.127.194:44707)
0923 20:21:35.189900 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.189909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.190277 (+ 368us) server_negotiation.cc:407] Connection header received
0923 20:21:35.190478 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.190487 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.190570 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.190718 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:35.191185 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.191949 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.193294 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.193784 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.193918 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.194326 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:35.194441 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:35.194531 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.194748 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.194827 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:21:35.195226 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.189622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44707 (local address 127.14.127.193:46093)
0923 20:21:35.190177 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:35.190192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:35.190346 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:35.190736 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:35.190741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:35.190752 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:21:35.191027 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.191036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.192090 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.192098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.193156 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.193167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.193950 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.193958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.194097 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.194147 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:21:35.194556 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:21:35.194574 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 20:21:35.194714 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 20:21:35.194847 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":278,"threads_started":1}
I20260923 20:21:35.267175 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.249495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:40929 (local address 127.14.127.193:41651)
0923 20:21:35.249988 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.249994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.259337 (+ 9343us) server_negotiation.cc:407] Connection header received
0923 20:21:35.259556 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.259562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.259644 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.259721 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:35.260413 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.261865 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.264711 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.265611 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.265713 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.266367 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:35.266520 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:35.266577 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.266811 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.266894 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260923 20:21:35.268040 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.249082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:41651 (local address 127.14.127.194:40929)
0923 20:21:35.259212 (+ 10130us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:35.259229 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:35.259414 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:35.259874 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:35.259880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:35.259892 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:21:35.260239 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.260249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.262039 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.262044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.264552 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:35.264566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.265913 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.265920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:35.266102 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.266174 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:21:35.266684 (+ 510us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:21:35.266693 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:21:35.267468 (+ 775us) client_negotiation.cc:240] Negotiation successful
0923 20:21:35.267624 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9922,"thread_start_us":9736,"threads_started":1}
I20260923 20:21:35.309499 15972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:35.309762 15972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:35.313238 15972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1b4fc6fcc2c4916a007ce2e09ddd3fb: Sending DeleteTablet for 3 replicas of tablet 38660c54a1444fd9a2900aa521830b98
I20260923 20:21:35.314016 16110 tablet_service.cc:1514] Processing DeleteTablet for tablet 38660c54a1444fd9a2900aa521830b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:35 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:58580
I20260923 20:21:35.314594 16248 tablet_service.cc:1514] Processing DeleteTablet for tablet 38660c54a1444fd9a2900aa521830b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:35 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:46686
I20260923 20:21:35.314602 16387 tablet_service.cc:1514] Processing DeleteTablet for tablet 38660c54a1444fd9a2900aa521830b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:35 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:44550
I20260923 20:21:35.324275 16548 tablet_replica.cc:330] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: stopping tablet replica
I20260923 20:21:35.324527 16548 raft_consensus.cc:2229] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:21:35.325198 16548 raft_consensus.cc:2258] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:35.328092 16548 ts_tablet_manager.cc:1905] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:35.332413 16550 tablet_replica.cc:330] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: stopping tablet replica
I20260923 20:21:35.332638 16550 raft_consensus.cc:2229] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:35.332895 16550 raft_consensus.cc:2258] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:35.334019 16550 ts_tablet_manager.cc:1905] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:35.341804 16549 tablet_replica.cc:330] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: stopping tablet replica
I20260923 20:21:35.342061 16550 ts_tablet_manager.cc:1918] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:35.342312 16550 log.cc:1196] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/38660c54a1444fd9a2900aa521830b98
I20260923 20:21:35.343001 16550 ts_tablet_manager.cc:1939] T 38660c54a1444fd9a2900aa521830b98 P 059ca99ca2a04940a7830c9b074cf00e: Deleting consensus metadata
I20260923 20:21:35.343024 16549 raft_consensus.cc:2229] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:35.343777 16549 raft_consensus.cc:2258] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:35.344412 15959 catalog_manager.cc:4841] TS 059ca99ca2a04940a7830c9b074cf00e (127.14.127.195:41467): tablet 38660c54a1444fd9a2900aa521830b98 (table test-table [id=e57bc47bcf3a4c76a8d12d3cc7ae6f8d]) successfully deleted
I20260923 20:21:35.345165 16549 ts_tablet_manager.cc:1905] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:35.351948 16548 ts_tablet_manager.cc:1918] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:35.352110 16548 log.cc:1196] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/38660c54a1444fd9a2900aa521830b98
I20260923 20:21:35.352569 16548 ts_tablet_manager.cc:1939] T 38660c54a1444fd9a2900aa521830b98 P eeb6fa0551004e899022ab6923751761: Deleting consensus metadata
I20260923 20:21:35.353668 15957 catalog_manager.cc:4841] TS eeb6fa0551004e899022ab6923751761 (127.14.127.193:41651): tablet 38660c54a1444fd9a2900aa521830b98 (table test-table [id=e57bc47bcf3a4c76a8d12d3cc7ae6f8d]) successfully deleted
I20260923 20:21:35.354862 16549 ts_tablet_manager.cc:1918] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:35.355008 16549 log.cc:1196] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/38660c54a1444fd9a2900aa521830b98
I20260923 20:21:35.355472 16549 ts_tablet_manager.cc:1939] T 38660c54a1444fd9a2900aa521830b98 P 57923a2b414944ae8939894ae135dbac: Deleting consensus metadata
I20260923 20:21:35.356565 15959 catalog_manager.cc:4841] TS 57923a2b414944ae8939894ae135dbac (127.14.127.194:44707): tablet 38660c54a1444fd9a2900aa521830b98 (table test-table [id=e57bc47bcf3a4c76a8d12d3cc7ae6f8d]) successfully deleted
Sep 23 20:21:35 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:21:35.369278 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.347736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.14.127.193:41651)
0923 20:21:35.347900 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.347905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.348878 (+ 973us) server_negotiation.cc:407] Connection header received
0923 20:21:35.349178 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.349183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.349273 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.349348 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:35.351475 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.352676 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.354770 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.355472 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.355568 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.361492 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:35.361512 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:35.361533 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:35.361581 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:35.365794 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.366721 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.366730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.366734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.366898 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.367330 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.367335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.367339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.367868 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:35.368201 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.368502 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.368756 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 20:21:35.370476 16175 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58604
I20260923 20:21:35.401748 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.374718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.14.127.254:36945)
0923 20:21:35.383569 (+ 8851us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.383576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.383599 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:21:35.383689 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.383694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.383802 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.386726 (+ 2924us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:35.386890 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.388104 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.389957 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.390659 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.390789 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.393785 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:35.393792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:35.393797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:35.393837 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:35.398493 (+ 4656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.399352 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.399358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.399362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.399584 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.400040 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.400046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.400049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.400502 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:35.400741 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.401110 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.401346 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8701,"thread_start_us":8403,"threads_started":1}
W20260923 20:21:35.402340 15972 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33326
I20260923 20:21:35.405092 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16044
I20260923 20:21:35.426807 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.406252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.14.127.193:41651)
0923 20:21:35.406427 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.406433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.411135 (+ 4702us) server_negotiation.cc:407] Connection header received
0923 20:21:35.411278 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.411284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.411399 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.411508 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:35.413031 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.414248 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.416037 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.416741 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.416867 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.419764 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:35.419771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:35.419776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:35.419818 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:35.423640 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.424478 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.424484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.424489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.424651 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.425063 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.425068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.425071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.425442 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:35.425744 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.426048 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.426315 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 20:21:35.427556 16175 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58606
W20260923 20:21:35.428189 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16044: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:35.428382 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 16044
I20260923 20:21:35.446545 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16184
Sep 23 20:21:35 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:21:35.463825 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.447454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.14.127.194:44707)
0923 20:21:35.447637 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.447644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.447668 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:21:35.447849 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.447854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.447933 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.448056 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:35.449245 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.450390 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.451725 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.452415 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.452574 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.456716 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:35.456730 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:35.456750 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:35.456808 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:35.460813 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.461515 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.461524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.461528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.461682 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.462093 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.462098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.462102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.462585 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:35.462952 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.463219 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.463442 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 20:21:35.464772 16314 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46700
W20260923 20:21:35.465459 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16184: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:35.465641 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 16184
I20260923 20:21:35.485056 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16323
Sep 23 20:21:35 dist-test-slave-wmd0 krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:21:35.500921 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:35.485771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.14.127.195:41467)
0923 20:21:35.486009 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:35.486014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:35.486034 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:21:35.486183 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:35.486195 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:35.486255 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:35.486358 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:35.487680 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.488375 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.489616 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:35.490024 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:35.490153 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:35.494960 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:35.494989 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:35.495005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:35.495044 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:35.497817 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.498571 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.498579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.498582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.498954 (+ 372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:35.499376 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:35.499381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:35.499384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:35.499756 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:35.500097 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:35.500326 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:21:35.500552 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 20:21:35.501677 16453 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44566
W20260923 20:21:35.502444 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16323: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:35.502583 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 16323
I20260923 20:21:35.523429 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 15941
I20260923 20:21:35.524636 16013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:35.639695 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 15941
2026-09-23T20:21:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4733 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:35 dist-test-slave-wmd0 krb5kdc[16560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:35 dist-test-slave-wmd0 krb5kdc[16560](info): set up 2 sockets
Sep 23 20:21:35 dist-test-slave-wmd0 krb5kdc[16560](info): commencing operation
krb5kdc: starting...
W20260923 20:21:36.718876 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:21:36 dist-test-slave-wmd0 krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:36.901549 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:42555
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:42555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:37.214656 16573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:37.215006 16573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:37.215170 16573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:37.222424 16573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:37.222572 16573 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:21:37.222683 16573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:37.222807 16573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:37.222892 16573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:37.222981 16573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:37.232693 16573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:42555
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:37.233974 16573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:37.235327 16573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:37.247218 16579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:37.247339 16578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:37.247298 16581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:37.248348 16573 server_base.cc:1034] running on GCE node
I20260923 20:21:37.249069 16573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:37.250394 16573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:37.251685 16573 hybrid_clock.cc:648] HybridClock initialized: now 1790194897251614 us; error 64 us; skew 500 ppm
I20260923 20:21:37.252053 16573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:37 dist-test-slave-wmd0 krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:37.255089 16573 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:37.256575 16573 webserver.cc:460] Webserver started at http://127.14.127.254:42399/ using document root <none> and password file <none>
I20260923 20:21:37.257277 16573 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:37.257404 16573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:37.257721 16573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:37.260764 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "ad54b7dfca3540ac92a188ac7ab28ba5"
format_stamp: "Formatted at 2026-09-23 20:21:37 on dist-test-slave-wmd0"
server_key: "5395e4dea1a5bce3c620a6e70c3c19d1"
server_key_iv: "530beea5a897253592fd38a605bb3ca8"
server_key_version: "encryptionkey@0"
I20260923 20:21:37.261565 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "ad54b7dfca3540ac92a188ac7ab28ba5"
format_stamp: "Formatted at 2026-09-23 20:21:37 on dist-test-slave-wmd0"
server_key: "5395e4dea1a5bce3c620a6e70c3c19d1"
server_key_iv: "530beea5a897253592fd38a605bb3ca8"
server_key_version: "encryptionkey@0"
I20260923 20:21:37.266748 16573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:21:37.270160 16588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:37.271515 16573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:37.271696 16573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "ad54b7dfca3540ac92a188ac7ab28ba5"
format_stamp: "Formatted at 2026-09-23 20:21:37 on dist-test-slave-wmd0"
server_key: "5395e4dea1a5bce3c620a6e70c3c19d1"
server_key_iv: "530beea5a897253592fd38a605bb3ca8"
server_key_version: "encryptionkey@0"
I20260923 20:21:37.271909 16573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:37.295253 16573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:37.299293 16573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:37.299577 16573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:37.321000 16573 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:42555
I20260923 20:21:37.321026 16649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:42555 every 8 connection(s)
I20260923 20:21:37.322348 16573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:37.325990 16650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:37.328516 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 16573
I20260923 20:21:37.328729 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:37.329502 14847 external_mini_cluster.cc:1428] Setting key 79bfcef48b8f96c9ec0a8ccd261633fb
I20260923 20:21:37.338757 16650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: Bootstrap starting.
I20260923 20:21:37.342572 16650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:37.343852 16650 log.cc:824] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:21:37 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:37.346858 16650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: No bootstrap required, opened a new log
I20260923 20:21:37.352123 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:37.331646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.14.127.254:42555)
0923 20:21:37.332552 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:37.332563 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:37.332603 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:21:37.334204 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:37.334225 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:37.334602 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:37.334979 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:37.336654 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.337901 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.339595 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.340226 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.340364 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:37.345044 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:37.345051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:37.345066 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:37.345106 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:37.348164 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:37.348841 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:37.348848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:37.348853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:37.349021 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:37.349413 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:37.349416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:37.349419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:37.349914 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:37.350239 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:37.350726 (+ 487us) server_negotiation.cc:299] Negotiation successful
0923 20:21:37.351143 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":473,"threads_started":1}
I20260923 20:21:37.354518 16650 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad54b7dfca3540ac92a188ac7ab28ba5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42555 } }
I20260923 20:21:37.354980 16650 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:37.355082 16650 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad54b7dfca3540ac92a188ac7ab28ba5, State: Initialized, Role: FOLLOWER
I20260923 20:21:37.355551 16650 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad54b7dfca3540ac92a188ac7ab28ba5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42555 } }
I20260923 20:21:37.355748 16650 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:37.355839 16650 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:37.355950 16650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:37.357928 16650 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad54b7dfca3540ac92a188ac7ab28ba5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42555 } }
I20260923 20:21:37.358394 16650 leader_election.cc:302] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad54b7dfca3540ac92a188ac7ab28ba5; no voters:
I20260923 20:21:37.358914 16650 leader_election.cc:288] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:37.359071 16655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:37.360018 16655 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [term 1 LEADER]: Becoming Leader. State: Replica: ad54b7dfca3540ac92a188ac7ab28ba5, State: Running, Role: LEADER
I20260923 20:21:37.360577 16655 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad54b7dfca3540ac92a188ac7ab28ba5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42555 } }
I20260923 20:21:37.362108 16650 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:37.363519 16657 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad54b7dfca3540ac92a188ac7ab28ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad54b7dfca3540ac92a188ac7ab28ba5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42555 } } }
I20260923 20:21:37.363854 16657 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:37.363448 16656 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad54b7dfca3540ac92a188ac7ab28ba5. Latest consensus state: current_term: 1 leader_uuid: "ad54b7dfca3540ac92a188ac7ab28ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad54b7dfca3540ac92a188ac7ab28ba5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42555 } } }
I20260923 20:21:37.364122 16656 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:37.366027 16660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:37.371196 16660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:37.380125 16660 catalog_manager.cc:1347] Generated new cluster ID: 54c33f875ec14015b487c2ce112b26b7
I20260923 20:21:37.380257 16660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:37.401958 16660 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:37.403354 16660 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:37.422411 16660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: Generated new TSK 0
I20260923 20:21:37.423571 16660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:21:37.477698 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42555
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:37.800446 16676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:37.800765 16676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:37.800940 16676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:37.808450 16676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:37.808569 16676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:37.808717 16676 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:21:37.808797 16676 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:21:37.808939 16676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:21:37.818764 16676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:42555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:37.820072 16676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:37.821408 16676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:37.834311 16684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:37.834456 16682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:37.834417 16681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:37.835439 16676 server_base.cc:1034] running on GCE node
I20260923 20:21:37.836063 16676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:37.836989 16676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:37.838213 16676 hybrid_clock.cc:648] HybridClock initialized: now 1790194897838157 us; error 50 us; skew 500 ppm
I20260923 20:21:37.838546 16676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:37 dist-test-slave-wmd0 krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:37.841658 16676 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:21:37.843219 16676 webserver.cc:460] Webserver started at http://127.14.127.193:43853/ using document root <none> and password file <none>
I20260923 20:21:37.843899 16676 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:37.844045 16676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:37.844353 16676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:37.847280 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "1e85f99f835a42fdbaab20e9351a3cb3"
format_stamp: "Formatted at 2026-09-23 20:21:37 on dist-test-slave-wmd0"
server_key: "ff835a33f19e4945ac170a945194a34d"
server_key_iv: "17a9dc7f4aea47abf37bdcd9a32e3c5d"
server_key_version: "encryptionkey@0"
I20260923 20:21:37.848048 16676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "1e85f99f835a42fdbaab20e9351a3cb3"
format_stamp: "Formatted at 2026-09-23 20:21:37 on dist-test-slave-wmd0"
server_key: "ff835a33f19e4945ac170a945194a34d"
server_key_iv: "17a9dc7f4aea47abf37bdcd9a32e3c5d"
server_key_version: "encryptionkey@0"
I20260923 20:21:37.853020 16676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:21:37.856304 16691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:37.857577 16676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:21:37.857748 16676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "1e85f99f835a42fdbaab20e9351a3cb3"
format_stamp: "Formatted at 2026-09-23 20:21:37 on dist-test-slave-wmd0"
server_key: "ff835a33f19e4945ac170a945194a34d"
server_key_iv: "17a9dc7f4aea47abf37bdcd9a32e3c5d"
server_key_version: "encryptionkey@0"
I20260923 20:21:37.857945 16676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:37.878160 16676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:37.882190 16676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:37.882471 16676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:37.885214 16676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:37.885339 16676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:37.885522 16676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:37.885613 16676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:21:37 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:37.908196 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:37.885682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:43473 (local address 127.14.127.254:42555)
0923 20:21:37.886552 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:37.886559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:37.887256 (+ 697us) server_negotiation.cc:407] Connection header received
0923 20:21:37.889191 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:37.889196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:37.889283 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:37.889413 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:37.891852 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.893190 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.895813 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.896266 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.896420 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:37.901720 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:37.901728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:37.901734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:37.901778 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:37.905130 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:37.905843 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:37.905849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:37.905852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:37.905972 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:37.906411 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:37.906416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:37.906420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:37.906780 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:37.906928 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:37.907625 (+ 697us) server_negotiation.cc:299] Negotiation successful
0923 20:21:37.907838 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":1015,"threads_started":1}
I20260923 20:21:37.909198 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:37.886273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42555 (local address 127.14.127.193:43473)
0923 20:21:37.887100 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:37.887138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:37.888918 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:37.889675 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:37.889690 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:37.890490 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:37.891546 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:37.891569 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.893383 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.893393 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:37.895651 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:37.895664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.896564 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.896577 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:37.896956 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:37.897783 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:37.897829 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:37.901368 (+ 3539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:37.905273 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:37.905282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:37.905298 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:37.905713 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:37.906088 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:37.906093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:37.906097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:37.906291 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:37.906923 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:37.906931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:37.907265 (+ 334us) client_negotiation.cc:769] Sending connection context
0923 20:21:37.907532 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 20:21:37.908022 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":370,"threads_started":1}
I20260923 20:21:37.931577 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:37.915671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:48327 (local address 127.14.127.254:42555)
0923 20:21:37.915892 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:37.915900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:37.916517 (+ 617us) server_negotiation.cc:407] Connection header received
0923 20:21:37.916645 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:37.916652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:37.916755 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:37.916886 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:37.917746 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.919161 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.921630 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.922050 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.922220 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:37.925306 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:37.925311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:37.925314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:37.925344 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:37.928225 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:37.929346 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:37.929354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:37.929358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:37.929530 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:37.929992 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:37.929999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:37.930006 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:37.930365 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:37.930530 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:37.930986 (+ 456us) server_negotiation.cc:299] Negotiation successful
0923 20:21:37.931228 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:21:37.931699 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:37.915790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42555 (local address 127.14.127.193:48327)
0923 20:21:37.916415 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:37.916431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:37.916543 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:37.916899 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:37.916902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:37.917333 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:37.917605 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:37.917613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.919343 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.919350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:37.921393 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:37.921428 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:37.922282 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:37.922292 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:37.922560 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:37.923420 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:37.923456 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:37.925035 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:37.928391 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:37.928407 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:37.928415 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:37.929156 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:37.929680 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:37.929687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:37.929696 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:37.929860 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:37.930482 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:37.930485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:37.930718 (+ 233us) client_negotiation.cc:769] Sending connection context
0923 20:21:37.930875 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 20:21:37.931244 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":336,"threads_started":1}
I20260923 20:21:37.941572 16676 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:34177
I20260923 20:21:37.941653 16811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:34177 every 8 connection(s)
I20260923 20:21:37.942868 16676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:21:37.944875 16812 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42555
I20260923 20:21:37.945106 16812 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:37.945619 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 16676
I20260923 20:21:37.945781 16812 heartbeater.cc:511] Master 127.14.127.254:42555 requested a full tablet report, sending...
I20260923 20:21:37.945815 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:21:37.946460 14847 external_mini_cluster.cc:1428] Setting key d5a97019dbb4636f863d20be7bbe8967
I20260923 20:21:37.947772 16605 ts_manager.cc:194] Registered new tserver with Master: 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177)
I20260923 20:21:37.949157 16605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:43473
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:21:37.987301 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42555
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:38.320927 16816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:38.321177 16816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:38.321341 16816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:38.328575 16816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:38.328678 16816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:38.328820 16816 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:21:38.328879 16816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:21:38.329010 16816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:21:38.338558 16816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:42555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:38.339885 16816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:38.341166 16816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:38.353832 16824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:38.353879 16821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:38.353843 16822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:38.354672 16816 server_base.cc:1034] running on GCE node
I20260923 20:21:38.355285 16816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:38.356213 16816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:38.357440 16816 hybrid_clock.cc:648] HybridClock initialized: now 1790194898357387 us; error 41 us; skew 500 ppm
I20260923 20:21:38.357828 16816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:38 dist-test-slave-wmd0 krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:38.361923 16816 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:21:38.363653 16816 webserver.cc:460] Webserver started at http://127.14.127.194:40381/ using document root <none> and password file <none>
I20260923 20:21:38.364276 16816 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:38.364375 16816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:38.364645 16816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:38.367547 16816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "392a7502d1ce489881076810bd559790"
format_stamp: "Formatted at 2026-09-23 20:21:38 on dist-test-slave-wmd0"
server_key: "d8091860ead6dc7e8ea17a16d59c925f"
server_key_iv: "adb5d70b8a3690b43fc3bb1897ef7c58"
server_key_version: "encryptionkey@0"
I20260923 20:21:38.368288 16816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "392a7502d1ce489881076810bd559790"
format_stamp: "Formatted at 2026-09-23 20:21:38 on dist-test-slave-wmd0"
server_key: "d8091860ead6dc7e8ea17a16d59c925f"
server_key_iv: "adb5d70b8a3690b43fc3bb1897ef7c58"
server_key_version: "encryptionkey@0"
I20260923 20:21:38.373322 16816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:21:38.376829 16831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:38.378127 16816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:21:38.378257 16816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "392a7502d1ce489881076810bd559790"
format_stamp: "Formatted at 2026-09-23 20:21:38 on dist-test-slave-wmd0"
server_key: "d8091860ead6dc7e8ea17a16d59c925f"
server_key_iv: "adb5d70b8a3690b43fc3bb1897ef7c58"
server_key_version: "encryptionkey@0"
I20260923 20:21:38.378414 16816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:38.390756 16816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:38.394338 16816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:38.394573 16816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:38.397030 16816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:38.397168 16816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:38.397284 16816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:38.397367 16816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:21:38 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:38.422338 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.397857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:34979 (local address 127.14.127.254:42555)
0923 20:21:38.398062 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:38.398067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:38.399395 (+ 1328us) server_negotiation.cc:407] Connection header received
0923 20:21:38.401621 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:38.401627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:38.401724 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:38.401859 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:38.404704 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.405651 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.409033 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.409627 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.409760 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:38.415061 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:38.415071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:38.415081 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:38.415130 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:38.418485 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:38.419443 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:38.419447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:38.419449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:38.419561 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:38.420138 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:38.420145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:38.420152 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:38.420604 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:38.420785 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:38.421497 (+ 712us) server_negotiation.cc:299] Negotiation successful
0923 20:21:38.421801 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:21:38.423487 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.398419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42555 (local address 127.14.127.194:34979)
0923 20:21:38.399234 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:38.399279 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:38.401272 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:38.402342 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:38.402358 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:38.403389 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:38.404500 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:38.404531 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.405859 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.405868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:38.408864 (+ 2996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:38.408884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.410021 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.410026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:38.410232 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:38.411305 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:38.411346 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:38.414757 (+ 3411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:38.418644 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:38.418654 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:38.418672 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:38.419306 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:38.419672 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:38.419677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:38.419684 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:38.419974 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:38.420754 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:38.420763 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:38.421099 (+ 336us) client_negotiation.cc:769] Sending connection context
0923 20:21:38.421314 (+ 215us) client_negotiation.cc:240] Negotiation successful
0923 20:21:38.421889 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":360,"threads_started":1}
I20260923 20:21:38.444283 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.429268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42555 (local address 127.14.127.194:39457)
0923 20:21:38.429829 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:38.429853 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:38.429989 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:38.430372 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:38.430375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:38.430851 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:38.431450 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:38.431464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.432657 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.432663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:38.434456 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:38.434467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.435627 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.435636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:38.435857 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:38.436660 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:38.436689 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:38.437883 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:38.441302 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:38.441309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:38.441313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:38.441975 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:38.442363 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:38.442367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:38.442375 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:38.442548 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:38.443396 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:38.443412 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:38.443608 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 20:21:38.443731 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 20:21:38.443962 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":378,"threads_started":1}
I20260923 20:21:38.444705 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.429581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:39457 (local address 127.14.127.254:42555)
0923 20:21:38.429816 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:38.429824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:38.429923 (+ 99us) server_negotiation.cc:407] Connection header received
0923 20:21:38.430127 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:38.430135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:38.430234 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:38.430362 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:38.431588 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.432523 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.434661 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.435187 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.435315 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:38.438120 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:38.438126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:38.438131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:38.438169 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:38.441171 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:38.442112 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:38.442120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:38.442124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:38.442252 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:38.442721 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:38.442729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:38.442733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:38.443244 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:38.443482 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:38.443909 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 20:21:38.444218 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:21:38.450496 16816 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:45549
I20260923 20:21:38.450556 16950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:45549 every 8 connection(s)
I20260923 20:21:38.451706 16816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:21:38.453665 16951 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42555
I20260923 20:21:38.453887 16951 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:38.454500 16951 heartbeater.cc:511] Master 127.14.127.254:42555 requested a full tablet report, sending...
I20260923 20:21:38.455305 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 16816
I20260923 20:21:38.455500 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:21:38.455890 16605 ts_manager.cc:194] Registered new tserver with Master: 392a7502d1ce489881076810bd559790 (127.14.127.194:45549)
I20260923 20:21:38.455941 14847 external_mini_cluster.cc:1428] Setting key f223324ac0fcf654a48b503cffb6b875
I20260923 20:21:38.456780 16605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:34979
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:21:38.495612 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42555
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:38.816351 16955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:38.816675 16955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:38.816856 16955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:38.823980 16955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:38.824134 16955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:38.824280 16955 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:21:38.824368 16955 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:21:38.824506 16955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:21:38.834739 16955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:42555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:38.836046 16955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:38.837385 16955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:38.850318 16961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:38.850296 16963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:38.850401 16960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:38.856084 16955 server_base.cc:1034] running on GCE node
I20260923 20:21:38.856722 16955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:38.857712 16955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:38.858906 16955 hybrid_clock.cc:648] HybridClock initialized: now 1790194898858866 us; error 28 us; skew 500 ppm
I20260923 20:21:38.859277 16955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:38 dist-test-slave-wmd0 krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:38.862473 16955 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:21:38.864101 16955 webserver.cc:460] Webserver started at http://127.14.127.195:36421/ using document root <none> and password file <none>
I20260923 20:21:38.864794 16955 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:38.864921 16955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:38.865237 16955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:38.868450 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "e5d11c10bf9e4d1395077f505fc30bf9"
format_stamp: "Formatted at 2026-09-23 20:21:38 on dist-test-slave-wmd0"
server_key: "0c61f33ec2fbfb5fdfb6ee590e5281c1"
server_key_iv: "633104b9195d262c6698e172ad0e238b"
server_key_version: "encryptionkey@0"
I20260923 20:21:38.869189 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "e5d11c10bf9e4d1395077f505fc30bf9"
format_stamp: "Formatted at 2026-09-23 20:21:38 on dist-test-slave-wmd0"
server_key: "0c61f33ec2fbfb5fdfb6ee590e5281c1"
server_key_iv: "633104b9195d262c6698e172ad0e238b"
server_key_version: "encryptionkey@0"
I20260923 20:21:38.874228 16955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 20:21:38.877601 16970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:38.878949 16955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:38.879123 16955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "e5d11c10bf9e4d1395077f505fc30bf9"
format_stamp: "Formatted at 2026-09-23 20:21:38 on dist-test-slave-wmd0"
server_key: "0c61f33ec2fbfb5fdfb6ee590e5281c1"
server_key_iv: "633104b9195d262c6698e172ad0e238b"
server_key_version: "encryptionkey@0"
I20260923 20:21:38.879316 16955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:38.892822 16955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:38.896539 16955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:38.896826 16955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:38.899438 16955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:38.899571 16955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:38.899701 16955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:38.899812 16955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:21:38 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:38.923903 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.899795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:41559 (local address 127.14.127.254:42555)
0923 20:21:38.900013 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:38.900021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:38.901523 (+ 1502us) server_negotiation.cc:407] Connection header received
0923 20:21:38.903613 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:38.903622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:38.903733 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:38.903860 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:38.906080 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.907218 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.909831 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.910572 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.910757 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:38.915889 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:38.915894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:38.915897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:38.915957 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:38.919374 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:38.920328 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:38.920333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:38.920337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:38.920503 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:38.921405 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:38.921412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:38.921418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:38.921845 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:38.922052 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:38.923057 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0923 20:21:38.923344 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:21:38.924659 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.900422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42555 (local address 127.14.127.195:41559)
0923 20:21:38.901340 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:38.901388 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:38.903327 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:38.904162 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:38.904173 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:38.904933 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:38.905874 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:38.905905 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.907387 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.907392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:38.909634 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:38.909653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.910872 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.910886 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:38.911186 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:38.911950 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:38.911994 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:38.915659 (+ 3665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:38.919508 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:38.919522 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:38.919544 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:38.920161 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:38.920703 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:38.920713 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:38.920720 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:38.921129 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:38.922033 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:38.922043 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:38.922488 (+ 445us) client_negotiation.cc:769] Sending connection context
0923 20:21:38.922905 (+ 417us) client_negotiation.cc:240] Negotiation successful
0923 20:21:38.923439 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":444,"threads_started":1}
I20260923 20:21:38.946841 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.930523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42555 (local address 127.14.127.195:45143)
0923 20:21:38.931269 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:38.931295 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:38.931481 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:38.931819 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:38.931826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:38.932526 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:38.932990 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:38.933007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.933944 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.933950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:38.936250 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:38.936269 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.937244 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.937254 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:38.937504 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:38.938308 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:38.938337 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:38.939515 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:38.943487 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:38.943491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:38.943494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:38.944254 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:38.944737 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:38.944744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:38.944751 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:38.944901 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:38.945584 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:38.945605 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:38.945841 (+ 236us) client_negotiation.cc:769] Sending connection context
0923 20:21:38.945990 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 20:21:38.946327 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":384,"threads_started":1}
I20260923 20:21:38.947714 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.930532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:45143 (local address 127.14.127.254:42555)
0923 20:21:38.930795 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:38.930804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:38.931393 (+ 589us) server_negotiation.cc:407] Connection header received
0923 20:21:38.931616 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:38.931620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:38.931676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:38.931754 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:38.933145 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.933798 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.936430 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:38.937017 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:38.937177 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:38.939718 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:38.939725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:38.939731 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:38.939775 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:38.943323 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:38.944416 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:38.944421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:38.944425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:38.944574 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:38.945047 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:38.945052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:38.945056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:38.945376 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:38.945515 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:38.947149 (+ 1634us) server_negotiation.cc:299] Negotiation successful
0923 20:21:38.947388 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:21:38.951916 16812 heartbeater.cc:503] Master 127.14.127.254:42555 was elected leader, sending a full tablet report...
I20260923 20:21:38.955267 16955 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:33937
I20260923 20:21:38.955332 17089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:33937 every 8 connection(s)
I20260923 20:21:38.956440 16955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:21:38.958231 17090 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42555
I20260923 20:21:38.958458 17090 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:38.959129 17090 heartbeater.cc:511] Master 127.14.127.254:42555 requested a full tablet report, sending...
I20260923 20:21:38.960341 16604 ts_manager.cc:194] Registered new tserver with Master: e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937)
I20260923 20:21:38.961320 16604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:41559
I20260923 20:21:38.962286 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 16955
I20260923 20:21:38.962481 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:21:38.963073 14847 external_mini_cluster.cc:1428] Setting key 264bd914e8d1d175f59cc4732478abeb
I20260923 20:21:38.965721 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:21:38 dist-test-slave-wmd0 krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:21:39 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:39.017056 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:38.997114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.14.127.254:42555)
0923 20:21:38.997354 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:38.997362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:38.997542 (+ 180us) server_negotiation.cc:407] Connection header received
0923 20:21:38.997768 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:38.997774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:38.997855 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:38.997964 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:38.999407 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.000141 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.001384 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.001805 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.001942 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.011848 (+ 9906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:39.011853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:39.011857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:39.011891 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:39.014405 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:39.015074 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:39.015081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:39.015085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:39.015217 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:39.015594 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:39.015599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:39.015602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:39.015872 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:39.016147 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.016463 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.016701 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:21:39.020879 16604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:21:39.028110 16604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:39.078168 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.040936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.14.127.194:45549)
0923 20:21:39.064777 (+ 23841us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.064791 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.065967 (+ 1176us) server_negotiation.cc:407] Connection header received
0923 20:21:39.066109 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.066123 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.066490 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.066791 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:39.067962 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.069942 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.074249 (+ 4307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.076007 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.076188 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.076375 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.077662 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.077848 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23573,"thread_start_us":23876,"threads_started":1}
I20260923 20:21:39.078969 17105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.040289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:45549 (local address 127.0.0.1:43138)
0923 20:21:39.065577 (+ 25288us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.065615 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.065792 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.067111 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.067123 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.067166 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:39.067702 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.067719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.070152 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.070161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.073412 (+ 3251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.073435 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.077067 (+ 3632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.077079 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.077382 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.077441 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.078364 (+ 923us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.078560 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25112,"thread_start_us":33672,"threads_started":1}
I20260923 20:21:39.081923 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.039546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:34177 (local address 127.0.0.1:60690)
0923 20:21:39.041141 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.041176 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.041332 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.074703 (+ 33371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.074708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.074735 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:39.075139 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.075151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.077117 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.077122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.079227 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.079244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.081130 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.081136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.081303 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.081335 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.081441 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.081583 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"thread_start_us":1199,"threads_started":1}
I20260923 20:21:39.081980 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.039712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.14.127.193:34177)
0923 20:21:39.074003 (+ 34291us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.074012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.074044 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:21:39.074150 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.074159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.074427 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.075342 (+ 915us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:39.075544 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.076965 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.079423 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.080896 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.081060 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.081237 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.081505 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.081680 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34143,"thread_start_us":34005,"threads_started":1}
I20260923 20:21:39.082803 16881 tablet_service.cc:1467] Processing CreateTablet for tablet a438972466a04e02b692bff26f25761a (DEFAULT_TABLE table=test-table [id=3cfc9a46110145ec9fd872ca108eed7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:39.084373 16881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a438972466a04e02b692bff26f25761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.085784 17107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.041572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:33937 (local address 127.0.0.1:33186)
0923 20:21:39.077301 (+ 35729us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.077319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.077471 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.078672 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.078678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.078705 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:39.079405 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.079421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.081332 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.081341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.083459 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.083471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.085010 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.085016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.085196 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.085236 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.085348 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.085470 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35613,"thread_start_us":35444,"threads_started":1}
I20260923 20:21:39.085879 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.041998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.14.127.195:33937)
0923 20:21:39.076878 (+ 34880us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.076888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.077427 (+ 539us) server_negotiation.cc:407] Connection header received
0923 20:21:39.077653 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.077658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.077828 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.078048 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:39.079623 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.081177 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.083666 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.084812 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.085012 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.085172 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.085402 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.085556 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34717,"thread_start_us":34545,"threads_started":1}
I20260923 20:21:39.085783 16742 tablet_service.cc:1467] Processing CreateTablet for tablet a438972466a04e02b692bff26f25761a (DEFAULT_TABLE table=test-table [id=3cfc9a46110145ec9fd872ca108eed7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:39.087157 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a438972466a04e02b692bff26f25761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.089205 17020 tablet_service.cc:1467] Processing CreateTablet for tablet a438972466a04e02b692bff26f25761a (DEFAULT_TABLE table=test-table [id=3cfc9a46110145ec9fd872ca108eed7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:39.090389 17020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a438972466a04e02b692bff26f25761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.098956 17109 tablet_bootstrap.cc:492] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Bootstrap starting.
I20260923 20:21:39.100895 17110 tablet_bootstrap.cc:492] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Bootstrap starting.
I20260923 20:21:39.103008 17109 tablet_bootstrap.cc:654] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.103400 17111 tablet_bootstrap.cc:492] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Bootstrap starting.
I20260923 20:21:39.104434 17109 log.cc:824] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:39.104578 17110 tablet_bootstrap.cc:654] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.105721 17110 log.cc:824] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:39.106645 17111 tablet_bootstrap.cc:654] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.107571 17109 tablet_bootstrap.cc:492] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: No bootstrap required, opened a new log
I20260923 20:21:39.107836 17109 ts_tablet_manager.cc:1397] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 20:21:39.107882 17111 log.cc:824] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:39.108244 17110 tablet_bootstrap.cc:492] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: No bootstrap required, opened a new log
I20260923 20:21:39.108515 17110 ts_tablet_manager.cc:1397] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260923 20:21:39.110739 17111 tablet_bootstrap.cc:492] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: No bootstrap required, opened a new log
I20260923 20:21:39.111009 17111 ts_tablet_manager.cc:1397] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260923 20:21:39.115684 17110 raft_consensus.cc:348] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.116029 17110 raft_consensus.cc:374] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.116171 17110 raft_consensus.cc:729] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d11c10bf9e4d1395077f505fc30bf9, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.116812 17110 consensus_queue.cc:260] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.117861 17090 heartbeater.cc:503] Master 127.14.127.254:42555 was elected leader, sending a full tablet report...
I20260923 20:21:39.118372 17110 ts_tablet_manager.cc:1428] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260923 20:21:39.119131 17109 raft_consensus.cc:348] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.119576 17109 raft_consensus.cc:374] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.119693 17109 raft_consensus.cc:729] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392a7502d1ce489881076810bd559790, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.120273 17109 consensus_queue.cc:260] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.121310 16951 heartbeater.cc:503] Master 127.14.127.254:42555 was elected leader, sending a full tablet report...
I20260923 20:21:39.121922 17109 ts_tablet_manager.cc:1428] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260923 20:21:39.122748 17111 raft_consensus.cc:348] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.123129 17111 raft_consensus.cc:374] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.123250 17111 raft_consensus.cc:729] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e85f99f835a42fdbaab20e9351a3cb3, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.123840 17111 consensus_queue.cc:260] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.125556 17111 ts_tablet_manager.cc:1428] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260923 20:21:39.131978 17115 raft_consensus.cc:482] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:39.132238 17115 raft_consensus.cc:504] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.133672 17115 leader_election.cc:288] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177), 392a7502d1ce489881076810bd559790 (127.14.127.194:45549)
I20260923 20:21:39.140021 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.133848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:34177 (local address 127.14.127.195:58173)
0923 20:21:39.134017 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.134040 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.134161 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.134741 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.134746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.134798 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:39.134994 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.135008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.136142 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.136148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.138031 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.138042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.139269 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.139276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.139476 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.139511 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.139643 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.139781 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":48}
I20260923 20:21:39.140056 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.134176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:58173 (local address 127.14.127.193:34177)
0923 20:21:39.134376 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.134384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.134403 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:21:39.134466 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.134472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.134538 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.134660 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:39.135137 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.135999 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.138172 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.139130 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.139243 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.139288 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.139672 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.139814 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:21:39.140559 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.134082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:36881 (local address 127.14.127.194:45549)
0923 20:21:39.134256 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.134264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.134702 (+ 438us) server_negotiation.cc:407] Connection header received
0923 20:21:39.134853 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.134858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.134926 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.135023 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:39.135577 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.136407 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.138688 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.139587 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.139749 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.139812 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.140174 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.140306 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 20:21:39.141080 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.133930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:45549 (local address 127.14.127.195:36881)
0923 20:21:39.134584 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.134603 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.134732 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.135038 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.135041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.135081 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:39.135434 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.135449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.136543 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.136547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.138478 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.138495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.139808 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.139814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.139978 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.140008 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.140730 (+ 722us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.140829 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":50,"thread_start_us":352,"threads_started":1}
I20260923 20:21:39.140934 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a438972466a04e02b692bff26f25761a" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" is_pre_election: true
I20260923 20:21:39.141419 16767 raft_consensus.cc:2454] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 0.
I20260923 20:21:39.142014 16971 leader_election.cc:302] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, e5d11c10bf9e4d1395077f505fc30bf9; no voters:
I20260923 20:21:39.142015 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a438972466a04e02b692bff26f25761a" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790" is_pre_election: true
I20260923 20:21:39.142410 17115 raft_consensus.cc:2781] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:39.142405 16906 raft_consensus.cc:2454] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 0.
I20260923 20:21:39.142554 17115 raft_consensus.cc:482] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:39.142679 17115 raft_consensus.cc:3037] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.144846 17115 raft_consensus.cc:504] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.145434 17115 leader_election.cc:288] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 election: Requested vote from peers 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177), 392a7502d1ce489881076810bd559790 (127.14.127.194:45549)
I20260923 20:21:39.145792 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a438972466a04e02b692bff26f25761a" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e85f99f835a42fdbaab20e9351a3cb3"
I20260923 20:21:39.145905 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a438972466a04e02b692bff26f25761a" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790"
I20260923 20:21:39.145991 16767 raft_consensus.cc:3037] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.146127 16906 raft_consensus.cc:3037] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.148190 16767 raft_consensus.cc:2454] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 1.
I20260923 20:21:39.148348 16906 raft_consensus.cc:2454] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 1.
I20260923 20:21:39.148581 16971 leader_election.cc:302] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, e5d11c10bf9e4d1395077f505fc30bf9; no voters:
I20260923 20:21:39.148849 17115 raft_consensus.cc:2781] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:39.149319 17115 raft_consensus.cc:686] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 LEADER]: Becoming Leader. State: Replica: e5d11c10bf9e4d1395077f505fc30bf9, State: Running, Role: LEADER
I20260923 20:21:39.149829 17115 consensus_queue.cc:237] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } }
I20260923 20:21:39.154011 16604 catalog_manager.cc:5495] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:21:39 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
W20260923 20:21:39.195926 16813 tablet.cc:2369] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:39.203619 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.185527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42555 (local address 127.0.0.1:59938)
0923 20:21:39.186124 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.186139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.186307 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.186906 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.186910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.187755 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:39.188131 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.188145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.189745 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.189748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.191373 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.191389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.192570 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.192582 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.192775 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.193501 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:39.193519 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:39.196276 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:39.200363 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:39.200393 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:39.200400 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:39.200974 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:39.201455 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:39.201460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:39.201466 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:39.201627 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:39.202402 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:39.202410 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:39.202661 (+ 251us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.202865 (+ 204us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.203184 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":306,"threads_started":1}
W20260923 20:21:39.203570 16952 tablet.cc:2369] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:39.203840 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.185465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.14.127.254:42555)
0923 20:21:39.185688 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.185697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.186226 (+ 529us) server_negotiation.cc:407] Connection header received
0923 20:21:39.186430 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.186436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.186530 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.186684 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:39.188307 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.189615 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.191535 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.192349 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.192489 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.196497 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:39.196505 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:39.196512 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:39.196553 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:39.200203 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:39.201108 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:39.201115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:39.201120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:39.201317 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:39.201772 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:39.201779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:39.201785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:39.202222 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:39.202405 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.202993 (+ 588us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.203368 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":51}
W20260923 20:21:39.207950 17091 tablet.cc:2369] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:39.208792 16604 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:59938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:21:39.209784 16604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:39.216464 16881 tablet_service.cc:1467] Processing CreateTablet for tablet 0410345787be466aab2bb85ee63d26de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=438d92e24cfd47e988f718d4c2264a33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:21:39.216281 16742 tablet_service.cc:1467] Processing CreateTablet for tablet 0410345787be466aab2bb85ee63d26de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=438d92e24cfd47e988f718d4c2264a33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:21:39.216576 17020 tablet_service.cc:1467] Processing CreateTablet for tablet 0410345787be466aab2bb85ee63d26de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=438d92e24cfd47e988f718d4c2264a33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:21:39.217280 16881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0410345787be466aab2bb85ee63d26de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.217280 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0410345787be466aab2bb85ee63d26de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.217326 17020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0410345787be466aab2bb85ee63d26de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.222927 17110 tablet_bootstrap.cc:492] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: Bootstrap starting.
I20260923 20:21:39.224298 17109 tablet_bootstrap.cc:492] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: Bootstrap starting.
I20260923 20:21:39.224413 17110 tablet_bootstrap.cc:654] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.224745 17111 tablet_bootstrap.cc:492] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: Bootstrap starting.
I20260923 20:21:39.225946 17109 tablet_bootstrap.cc:654] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.226544 17111 tablet_bootstrap.cc:654] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.229014 17110 tablet_bootstrap.cc:492] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: No bootstrap required, opened a new log
I20260923 20:21:39.229115 17109 tablet_bootstrap.cc:492] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: No bootstrap required, opened a new log
I20260923 20:21:39.229135 17110 ts_tablet_manager.cc:1397] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260923 20:21:39.229185 17111 tablet_bootstrap.cc:492] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: No bootstrap required, opened a new log
I20260923 20:21:39.229259 17109 ts_tablet_manager.cc:1397] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 20:21:39.229306 17111 ts_tablet_manager.cc:1397] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 20:21:39.229789 17110 raft_consensus.cc:348] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.229991 17110 raft_consensus.cc:374] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.230093 17110 raft_consensus.cc:729] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d11c10bf9e4d1395077f505fc30bf9, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.229991 17111 raft_consensus.cc:348] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.229991 17109 raft_consensus.cc:348] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.230237 17111 raft_consensus.cc:374] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.230237 17109 raft_consensus.cc:374] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.230337 17111 raft_consensus.cc:729] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e85f99f835a42fdbaab20e9351a3cb3, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.230337 17109 raft_consensus.cc:729] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392a7502d1ce489881076810bd559790, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.230335 17110 consensus_queue.cc:260] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.230561 17109 consensus_queue.cc:260] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.230561 17111 consensus_queue.cc:260] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.231201 17111 ts_tablet_manager.cc:1428] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 20:21:39.231199 17109 ts_tablet_manager.cc:1428] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:21:39.231230 17116 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.231631 17116 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:39.231604 17117 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.231956 17117 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:39.231964 17115 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.232242 17115 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:39.233633 17110 ts_tablet_manager.cc:1428] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260923 20:21:39.403483 17117 raft_consensus.cc:482] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:39.403724 17117 raft_consensus.cc:504] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.405350 17117 leader_election.cc:288] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 392a7502d1ce489881076810bd559790 (127.14.127.194:45549), e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937)
I20260923 20:21:39.412706 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.405846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:57773 (local address 127.14.127.194:45549)
0923 20:21:39.406106 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.406115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.407042 (+ 927us) server_negotiation.cc:407] Connection header received
0923 20:21:39.407220 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.407225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.407283 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.407412 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:39.408228 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.409216 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.410954 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.411720 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.411912 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.411994 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.412304 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.412428 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 20:21:39.412834 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.405674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:45549 (local address 127.14.127.193:57773)
0923 20:21:39.406942 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.406956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.407064 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.407426 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.407431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.407452 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:39.407963 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.407978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.409358 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.409367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.410796 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.410808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.411950 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.411956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.412142 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.412167 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.412275 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.412411 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":2,"thread_start_us":743,"threads_started":1}
I20260923 20:21:39.412986 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.405848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:50315 (local address 127.14.127.195:33937)
0923 20:21:39.406107 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.406115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.406929 (+ 814us) server_negotiation.cc:407] Connection header received
0923 20:21:39.407184 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.407189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.407250 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.407357 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:21:39.408134 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.409100 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.410953 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.411823 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.412018 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.412101 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.412415 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.412556 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 20:21:39.413173 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.405658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:33937 (local address 127.14.127.193:50315)
0923 20:21:39.406787 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.406818 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.407028 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.407393 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.407397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.407419 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:21:39.407908 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.407922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.409227 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.409235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.410761 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.410773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.412050 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.412056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.412267 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.412292 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.412391 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.412562 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":3,"thread_start_us":700,"threads_started":1}
I20260923 20:21:39.413601 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0410345787be466aab2bb85ee63d26de" candidate_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790" is_pre_election: true
I20260923 20:21:39.413882 16906 raft_consensus.cc:2454] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e85f99f835a42fdbaab20e9351a3cb3 in term 0.
I20260923 20:21:39.414245 17040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0410345787be466aab2bb85ee63d26de" candidate_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" is_pre_election: true
I20260923 20:21:39.414367 16694 leader_election.cc:302] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, 392a7502d1ce489881076810bd559790; no voters:
I20260923 20:21:39.414696 17040 raft_consensus.cc:2454] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e85f99f835a42fdbaab20e9351a3cb3 in term 0.
I20260923 20:21:39.414769 17117 raft_consensus.cc:2781] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:39.414885 17117 raft_consensus.cc:482] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:39.414986 17117 raft_consensus.cc:3037] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.417083 17117 raft_consensus.cc:504] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.417603 17117 leader_election.cc:288] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 392a7502d1ce489881076810bd559790 (127.14.127.194:45549), e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937)
I20260923 20:21:39.418052 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0410345787be466aab2bb85ee63d26de" candidate_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790"
I20260923 20:21:39.418080 17040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0410345787be466aab2bb85ee63d26de" candidate_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d11c10bf9e4d1395077f505fc30bf9"
I20260923 20:21:39.418251 16906 raft_consensus.cc:3037] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.418291 17040 raft_consensus.cc:3037] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.420321 16906 raft_consensus.cc:2454] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e85f99f835a42fdbaab20e9351a3cb3 in term 1.
I20260923 20:21:39.420471 17040 raft_consensus.cc:2454] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e85f99f835a42fdbaab20e9351a3cb3 in term 1.
I20260923 20:21:39.420727 16694 leader_election.cc:302] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, 392a7502d1ce489881076810bd559790; no voters:
I20260923 20:21:39.420986 17117 raft_consensus.cc:2781] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:39.421397 17117 raft_consensus.cc:686] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 1e85f99f835a42fdbaab20e9351a3cb3, State: Running, Role: LEADER
I20260923 20:21:39.421854 17117 consensus_queue.cc:237] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.425045 17132 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: TxnStatusTablet state changed. Reason: New leader 1e85f99f835a42fdbaab20e9351a3cb3. Latest consensus state: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.425452 17132 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:21:39.426174 17134 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:21:39.426455 17134 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:21:39.426496 16603 catalog_manager.cc:5495] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } health_report { overall_health: UNKNOWN } } }
I20260923 20:21:39.469117 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.462934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:34177 (local address 127.0.0.1:60706)
0923 20:21:39.463171 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.463189 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.463397 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.463978 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.463984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.463997 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:21:39.464195 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.464204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.465324 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.465332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.466336 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.466348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.467248 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.467254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.467553 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.467627 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:21:39.468483 (+ 856us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:21:39.468505 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.468579 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.468735 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 20:21:39.469120 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.462798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.14.127.193:34177)
0923 20:21:39.462979 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.462986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.463267 (+ 281us) server_negotiation.cc:407] Connection header received
0923 20:21:39.463637 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.463643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.463760 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.463944 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:39.464328 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.465185 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.466481 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.467062 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.467210 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.467821 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:39.468371 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:39.468616 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.468666 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.468776 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:21:39.476459 17040 raft_consensus.cc:1261] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Refusing update from remote peer 1e85f99f835a42fdbaab20e9351a3cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:39.476481 16906 raft_consensus.cc:1261] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Refusing update from remote peer 1e85f99f835a42fdbaab20e9351a3cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:39.477449 17117 consensus_queue.cc:1035] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:39.477789 17132 consensus_queue.cc:1035] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:39.484241 17115 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: TxnStatusTablet state changed. Reason: New leader 1e85f99f835a42fdbaab20e9351a3cb3. Latest consensus state: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.484543 17115 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:39.487015 17137 mvcc.cc:205] Tried to move back new op lower bound from 7332638308245938176 to 7332638308035678208. Current Snapshot: MvccSnapshot[applied={T|T < 7332638308245938176}]
I20260923 20:21:39.487910 17117 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.488209 17117 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:21:39.488776 17138 mvcc.cc:205] Tried to move back new op lower bound from 7332638308245938176 to 7332638308035678208. Current Snapshot: MvccSnapshot[applied={T|T < 7332638308245938176}]
I20260923 20:21:39.492585 17116 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: TxnStatusTablet state changed. Reason: New leader 1e85f99f835a42fdbaab20e9351a3cb3. Latest consensus state: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.492877 17116 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:39.496098 17115 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.496430 17115 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P e5d11c10bf9e4d1395077f505fc30bf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:39.509141 17132 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.509857 17132 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P 1e85f99f835a42fdbaab20e9351a3cb3: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:21:39.511201 17116 tablet_replica.cc:439] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } } }
I20260923 20:21:39.511453 17116 tablet_replica.cc:442] T 0410345787be466aab2bb85ee63d26de P 392a7502d1ce489881076810bd559790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:21:39.514103 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.504049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.14.127.195:33937)
0923 20:21:39.504274 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.504282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.504919 (+ 637us) server_negotiation.cc:407] Connection header received
0923 20:21:39.505049 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.505057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.505186 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.505358 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:39.506119 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.507680 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.510838 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.511443 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.512551 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.512698 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:39.513382 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:39.513518 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.513718 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.513800 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:21:39.530093 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.524660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:45307 (local address 127.14.127.193:34177)
0923 20:21:39.524875 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.524882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.525026 (+ 144us) server_negotiation.cc:407] Connection header received
0923 20:21:39.525249 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.525258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.525331 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.525429 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:39.526061 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.526926 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.528331 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.528761 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.528895 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.529353 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:39.529453 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:39.529557 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.529693 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.529768 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:21:39.530186 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.524341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:34177 (local address 127.14.127.195:45307)
0923 20:21:39.524911 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.524931 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.525099 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.525493 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.525500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.525513 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:21:39.525885 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.525894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.527050 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.527055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.528194 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.528206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.528959 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.528974 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.529105 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.529159 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:21:39.529559 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:21:39.529573 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.529672 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.529802 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":275,"threads_started":1}
I20260923 20:21:39.540390 16906 raft_consensus.cc:1261] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Refusing update from remote peer e5d11c10bf9e4d1395077f505fc30bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:39.540611 16767 raft_consensus.cc:1261] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Refusing update from remote peer e5d11c10bf9e4d1395077f505fc30bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:39.541121 17119 consensus_queue.cc:1035] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:39.541770 17115 consensus_queue.cc:1035] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:39.583984 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.577008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:53933 (local address 127.14.127.195:33937)
0923 20:21:39.577192 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:39.577199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:39.577869 (+ 670us) server_negotiation.cc:407] Connection header received
0923 20:21:39.578097 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:39.578101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:39.578161 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:39.578232 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:39.578917 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.579742 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.581848 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.582390 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.582532 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.583121 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:39.583256 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:39.583354 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:39.583552 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 20:21:39.583631 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 20:21:39.584343 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:39.576910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:33937 (local address 127.14.127.193:53933)
0923 20:21:39.577771 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:39.577791 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:39.577977 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:39.578279 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:39.578284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:39.578298 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:21:39.578735 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.578748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.579889 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.579895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.581698 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:39.581719 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:39.582598 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:39.582604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:39.582847 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:39.582925 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:21:39.583401 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:21:39.583414 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 20:21:39.583522 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:21:39.583661 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":501,"threads_started":1}
I20260923 20:21:39.626777 16604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:39.627079 16604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:39.631624 16604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: Sending DeleteTablet for 3 replicas of tablet a438972466a04e02b692bff26f25761a
I20260923 20:21:39.632941 16742 tablet_service.cc:1514] Processing DeleteTablet for tablet a438972466a04e02b692bff26f25761a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:39 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:60690
I20260923 20:21:39.632937 16881 tablet_service.cc:1514] Processing DeleteTablet for tablet a438972466a04e02b692bff26f25761a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:39 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43138
I20260923 20:21:39.633869 17020 tablet_service.cc:1514] Processing DeleteTablet for tablet a438972466a04e02b692bff26f25761a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:39 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:33186
I20260923 20:21:39.642055 16604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:21:39.642820 16604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:39.655879 16741 tablet_service.cc:1467] Processing CreateTablet for tablet f5af71b1630c4f0eb38ee6f1751bb526 (DEFAULT_TABLE table=test-table [id=cb24ac4ea7bb45a493677393b00a7b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:39.656499 16741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5af71b1630c4f0eb38ee6f1751bb526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.659644 16880 tablet_service.cc:1467] Processing CreateTablet for tablet f5af71b1630c4f0eb38ee6f1751bb526 (DEFAULT_TABLE table=test-table [id=cb24ac4ea7bb45a493677393b00a7b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:39.660197 16880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5af71b1630c4f0eb38ee6f1751bb526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.663422 17019 tablet_service.cc:1467] Processing CreateTablet for tablet f5af71b1630c4f0eb38ee6f1751bb526 (DEFAULT_TABLE table=test-table [id=cb24ac4ea7bb45a493677393b00a7b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:39.663978 17019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5af71b1630c4f0eb38ee6f1751bb526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:39.673003 17111 tablet_bootstrap.cc:492] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: Bootstrap starting.
I20260923 20:21:39.675537 17111 tablet_bootstrap.cc:654] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.687516 17110 tablet_bootstrap.cc:492] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: Bootstrap starting.
I20260923 20:21:39.689366 17110 tablet_bootstrap.cc:654] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.693336 17109 tablet_bootstrap.cc:492] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: Bootstrap starting.
I20260923 20:21:39.694574 17111 tablet_bootstrap.cc:492] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: No bootstrap required, opened a new log
I20260923 20:21:39.694854 17111 ts_tablet_manager.cc:1397] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.001s
I20260923 20:21:39.695199 17109 tablet_bootstrap.cc:654] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:39.695896 17111 raft_consensus.cc:348] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.696234 17111 raft_consensus.cc:374] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.696301 17111 raft_consensus.cc:729] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e85f99f835a42fdbaab20e9351a3cb3, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.696585 17111 consensus_queue.cc:260] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.698972 17111 ts_tablet_manager.cc:1428] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260923 20:21:39.720521 17109 tablet_bootstrap.cc:492] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: No bootstrap required, opened a new log
I20260923 20:21:39.720718 17109 ts_tablet_manager.cc:1397] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20260923 20:21:39.721529 17109 raft_consensus.cc:348] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.721797 17109 raft_consensus.cc:374] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.721877 17109 raft_consensus.cc:729] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392a7502d1ce489881076810bd559790, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.722193 17109 consensus_queue.cc:260] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.723843 17109 ts_tablet_manager.cc:1428] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:21:39.725872 17110 tablet_bootstrap.cc:492] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: No bootstrap required, opened a new log
I20260923 20:21:39.726058 17110 ts_tablet_manager.cc:1397] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.011s
I20260923 20:21:39.727490 17160 tablet_replica.cc:330] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: stopping tablet replica
I20260923 20:21:39.727914 17160 raft_consensus.cc:2229] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:39.728269 17160 raft_consensus.cc:2258] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:39.729167 17161 tablet_replica.cc:330] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: stopping tablet replica
I20260923 20:21:39.729643 17161 raft_consensus.cc:2229] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:21:39.731897 17160 ts_tablet_manager.cc:1905] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:39.735287 17159 tablet_replica.cc:330] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: stopping tablet replica
I20260923 20:21:39.735745 17159 raft_consensus.cc:2229] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:39.736120 17159 raft_consensus.cc:2258] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:39.737371 17110 raft_consensus.cc:348] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.737936 17110 raft_consensus.cc:374] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:39.738313 17110 raft_consensus.cc:729] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d11c10bf9e4d1395077f505fc30bf9, State: Initialized, Role: FOLLOWER
I20260923 20:21:39.738699 17159 ts_tablet_manager.cc:1905] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:39.738904 17161 raft_consensus.cc:2258] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:39.739233 17110 consensus_queue.cc:260] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.741119 17110 ts_tablet_manager.cc:1428] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20260923 20:21:39.743114 17161 ts_tablet_manager.cc:1905] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:39.748001 17160 ts_tablet_manager.cc:1918] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:39.748190 17160 log.cc:1196] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/a438972466a04e02b692bff26f25761a
I20260923 20:21:39.748710 17160 ts_tablet_manager.cc:1939] T a438972466a04e02b692bff26f25761a P 392a7502d1ce489881076810bd559790: Deleting consensus metadata
I20260923 20:21:39.750504 16591 catalog_manager.cc:4841] TS 392a7502d1ce489881076810bd559790 (127.14.127.194:45549): tablet a438972466a04e02b692bff26f25761a (table test-table [id=3cfc9a46110145ec9fd872ca108eed7a]) successfully deleted
I20260923 20:21:39.750972 17161 ts_tablet_manager.cc:1918] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:39.751123 17161 log.cc:1196] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/a438972466a04e02b692bff26f25761a
I20260923 20:21:39.751609 17161 ts_tablet_manager.cc:1939] T a438972466a04e02b692bff26f25761a P e5d11c10bf9e4d1395077f505fc30bf9: Deleting consensus metadata
I20260923 20:21:39.753170 16589 catalog_manager.cc:4841] TS e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937): tablet a438972466a04e02b692bff26f25761a (table test-table [id=3cfc9a46110145ec9fd872ca108eed7a]) successfully deleted
I20260923 20:21:39.757061 17159 ts_tablet_manager.cc:1918] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:39.757246 17159 log.cc:1196] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/a438972466a04e02b692bff26f25761a
I20260923 20:21:39.757820 17159 ts_tablet_manager.cc:1939] T a438972466a04e02b692bff26f25761a P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting consensus metadata
I20260923 20:21:39.759557 16589 catalog_manager.cc:4841] TS 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177): tablet a438972466a04e02b692bff26f25761a (table test-table [id=3cfc9a46110145ec9fd872ca108eed7a]) successfully deleted
I20260923 20:21:39.924784 17119 raft_consensus.cc:482] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:39.925079 17119 raft_consensus.cc:504] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.925967 17119 leader_election.cc:288] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177), 392a7502d1ce489881076810bd559790 (127.14.127.194:45549)
I20260923 20:21:39.926568 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5af71b1630c4f0eb38ee6f1751bb526" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" is_pre_election: true
I20260923 20:21:39.926654 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5af71b1630c4f0eb38ee6f1751bb526" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790" is_pre_election: true
I20260923 20:21:39.926898 16906 raft_consensus.cc:2454] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 0.
I20260923 20:21:39.926900 16767 raft_consensus.cc:2454] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 0.
I20260923 20:21:39.927352 16971 leader_election.cc:302] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, e5d11c10bf9e4d1395077f505fc30bf9; no voters:
I20260923 20:21:39.927768 17119 raft_consensus.cc:2781] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:39.927881 17119 raft_consensus.cc:482] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:39.927996 17119 raft_consensus.cc:3037] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.930832 17119 raft_consensus.cc:504] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.931470 17119 leader_election.cc:288] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 election: Requested vote from peers 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177), 392a7502d1ce489881076810bd559790 (127.14.127.194:45549)
I20260923 20:21:39.931913 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5af71b1630c4f0eb38ee6f1751bb526" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e85f99f835a42fdbaab20e9351a3cb3"
I20260923 20:21:39.932123 16767 raft_consensus.cc:3037] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.932076 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5af71b1630c4f0eb38ee6f1751bb526" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790"
I20260923 20:21:39.932289 16906 raft_consensus.cc:3037] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:39.934085 16767 raft_consensus.cc:2454] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 1.
I20260923 20:21:39.934475 16971 leader_election.cc:302] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, e5d11c10bf9e4d1395077f505fc30bf9; no voters:
I20260923 20:21:39.934816 17119 raft_consensus.cc:2781] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:39.935010 17119 raft_consensus.cc:686] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 LEADER]: Becoming Leader. State: Replica: e5d11c10bf9e4d1395077f505fc30bf9, State: Running, Role: LEADER
I20260923 20:21:39.935375 16906 raft_consensus.cc:2454] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 1.
I20260923 20:21:39.935294 17119 consensus_queue.cc:237] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:39.938246 16604 catalog_manager.cc:5495] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } health_report { overall_health: HEALTHY } } }
I20260923 20:21:40.008718 16905 raft_consensus.cc:1261] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Refusing update from remote peer e5d11c10bf9e4d1395077f505fc30bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:40.008998 16767 raft_consensus.cc:1261] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Refusing update from remote peer e5d11c10bf9e4d1395077f505fc30bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:40.009402 17119 consensus_queue.cc:1035] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:40.009713 17115 consensus_queue.cc:1035] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:21:40.038501 17138 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 20:21:41.626438 16604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:41.626739 16604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:41.632018 16604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: Sending DeleteTablet for 3 replicas of tablet f5af71b1630c4f0eb38ee6f1751bb526
I20260923 20:21:41.632746 16742 tablet_service.cc:1514] Processing DeleteTablet for tablet f5af71b1630c4f0eb38ee6f1751bb526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:60690
I20260923 20:21:41.632905 16881 tablet_service.cc:1514] Processing DeleteTablet for tablet f5af71b1630c4f0eb38ee6f1751bb526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43138
I20260923 20:21:41.632937 17020 tablet_service.cc:1514] Processing DeleteTablet for tablet f5af71b1630c4f0eb38ee6f1751bb526 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:33186
I20260923 20:21:41.633571 17191 tablet_replica.cc:330] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: stopping tablet replica
I20260923 20:21:41.633742 17190 tablet_replica.cc:330] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: stopping tablet replica
I20260923 20:21:41.633776 17191 raft_consensus.cc:2229] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:41.633876 17190 raft_consensus.cc:2229] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:41.634009 17191 raft_consensus.cc:2258] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:41.634096 17190 raft_consensus.cc:2258] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:41.635040 17190 ts_tablet_manager.cc:1905] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:41.635540 17191 ts_tablet_manager.cc:1905] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:41.641721 17190 ts_tablet_manager.cc:1918] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:21:41.641873 17190 log.cc:1196] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/f5af71b1630c4f0eb38ee6f1751bb526
I20260923 20:21:41.642074 17191 ts_tablet_manager.cc:1918] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:21:41.642263 17191 log.cc:1196] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/f5af71b1630c4f0eb38ee6f1751bb526
I20260923 20:21:41.642383 17190 ts_tablet_manager.cc:1939] T f5af71b1630c4f0eb38ee6f1751bb526 P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting consensus metadata
I20260923 20:21:41.642501 17192 tablet_replica.cc:330] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: stopping tablet replica
I20260923 20:21:41.642787 17192 raft_consensus.cc:2229] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:21:41.642820 17191 ts_tablet_manager.cc:1939] T f5af71b1630c4f0eb38ee6f1751bb526 P 392a7502d1ce489881076810bd559790: Deleting consensus metadata
I20260923 20:21:41.643215 17192 raft_consensus.cc:2258] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:41.643674 16589 catalog_manager.cc:4841] TS 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177): tablet f5af71b1630c4f0eb38ee6f1751bb526 (table test-table [id=cb24ac4ea7bb45a493677393b00a7b97]) successfully deleted
I20260923 20:21:41.644029 16591 catalog_manager.cc:4841] TS 392a7502d1ce489881076810bd559790 (127.14.127.194:45549): tablet f5af71b1630c4f0eb38ee6f1751bb526 (table test-table [id=cb24ac4ea7bb45a493677393b00a7b97]) successfully deleted
I20260923 20:21:41.644318 17192 ts_tablet_manager.cc:1905] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:41.652623 17192 ts_tablet_manager.cc:1918] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:21:41.652853 17192 log.cc:1196] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/f5af71b1630c4f0eb38ee6f1751bb526
I20260923 20:21:41.653308 17192 ts_tablet_manager.cc:1939] T f5af71b1630c4f0eb38ee6f1751bb526 P e5d11c10bf9e4d1395077f505fc30bf9: Deleting consensus metadata
I20260923 20:21:41.654223 16589 catalog_manager.cc:4841] TS e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937): tablet f5af71b1630c4f0eb38ee6f1751bb526 (table test-table [id=cb24ac4ea7bb45a493677393b00a7b97]) successfully deleted
Sep 23 20:21:41 dist-test-slave-wmd0 krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:21:41 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:41.691330 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:41.672793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.14.127.254:42555)
0923 20:21:41.673488 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:41.673494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:41.673517 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:21:41.673731 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:41.673735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:41.673798 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:41.673911 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:41.675749 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.676613 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.677800 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.678242 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.678394 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:41.686026 (+ 7632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:41.686031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:41.686034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:41.686068 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:41.688904 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:41.689493 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:41.689497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:41.689499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:41.689610 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:41.689951 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:41.689954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:41.689955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:41.690308 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:41.690492 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:41.690786 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 20:21:41.690961 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":389,"threads_started":1}
I20260923 20:21:41.694783 16604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:21:41.695318 16604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:41.701357 16742 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5d64c7f14e45e5af765bd5c27c858b (DEFAULT_TABLE table=test-table [id=d425457378754a89b43fbbbad1a00873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:41.701682 17020 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5d64c7f14e45e5af765bd5c27c858b (DEFAULT_TABLE table=test-table [id=d425457378754a89b43fbbbad1a00873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:41.701812 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5d64c7f14e45e5af765bd5c27c858b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:41.701705 16881 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5d64c7f14e45e5af765bd5c27c858b (DEFAULT_TABLE table=test-table [id=d425457378754a89b43fbbbad1a00873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:41.702067 17020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5d64c7f14e45e5af765bd5c27c858b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:41.702154 16881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5d64c7f14e45e5af765bd5c27c858b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:41.709005 17201 tablet_bootstrap.cc:492] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: Bootstrap starting.
I20260923 20:21:41.711339 17201 tablet_bootstrap.cc:654] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:41.713297 17202 tablet_bootstrap.cc:492] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: Bootstrap starting.
I20260923 20:21:41.713871 17201 tablet_bootstrap.cc:492] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: No bootstrap required, opened a new log
I20260923 20:21:41.713999 17201 ts_tablet_manager.cc:1397] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 20:21:41.714257 17203 tablet_bootstrap.cc:492] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: Bootstrap starting.
I20260923 20:21:41.715018 17202 tablet_bootstrap.cc:654] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:41.714978 17201 raft_consensus.cc:348] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.715279 17201 raft_consensus.cc:374] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:41.715420 17201 raft_consensus.cc:729] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392a7502d1ce489881076810bd559790, State: Initialized, Role: FOLLOWER
I20260923 20:21:41.715826 17203 tablet_bootstrap.cc:654] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:41.715747 17201 consensus_queue.cc:260] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.716797 17201 ts_tablet_manager.cc:1428] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:21:41.717540 17202 tablet_bootstrap.cc:492] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: No bootstrap required, opened a new log
I20260923 20:21:41.717681 17202 ts_tablet_manager.cc:1397] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:21:41.718429 17202 raft_consensus.cc:348] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.718676 17202 raft_consensus.cc:374] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:41.718792 17202 raft_consensus.cc:729] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e85f99f835a42fdbaab20e9351a3cb3, State: Initialized, Role: FOLLOWER
I20260923 20:21:41.719050 17202 consensus_queue.cc:260] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.719655 17202 ts_tablet_manager.cc:1428] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:21:41.721247 17203 tablet_bootstrap.cc:492] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: No bootstrap required, opened a new log
I20260923 20:21:41.721398 17203 ts_tablet_manager.cc:1397] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260923 20:21:41.722211 17203 raft_consensus.cc:348] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.722385 17203 raft_consensus.cc:374] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:41.722494 17203 raft_consensus.cc:729] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d11c10bf9e4d1395077f505fc30bf9, State: Initialized, Role: FOLLOWER
I20260923 20:21:41.722779 17203 consensus_queue.cc:260] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.723380 17203 ts_tablet_manager.cc:1428] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:21:41.729122 17132 raft_consensus.cc:482] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:41.729267 17132 raft_consensus.cc:504] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.729765 17132 leader_election.cc:288] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 392a7502d1ce489881076810bd559790 (127.14.127.194:45549), e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937)
I20260923 20:21:41.730178 16905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d64c7f14e45e5af765bd5c27c858b" candidate_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790" is_pre_election: true
I20260923 20:21:41.730239 17040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d64c7f14e45e5af765bd5c27c858b" candidate_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" is_pre_election: true
I20260923 20:21:41.730412 16905 raft_consensus.cc:2454] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e85f99f835a42fdbaab20e9351a3cb3 in term 0.
I20260923 20:21:41.730464 17040 raft_consensus.cc:2454] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e85f99f835a42fdbaab20e9351a3cb3 in term 0.
I20260923 20:21:41.730902 16694 leader_election.cc:302] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, 392a7502d1ce489881076810bd559790; no voters:
I20260923 20:21:41.731195 17132 raft_consensus.cc:2781] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:41.731276 17132 raft_consensus.cc:482] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:41.731393 17132 raft_consensus.cc:3037] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:41.733393 17132 raft_consensus.cc:504] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.734079 17132 leader_election.cc:288] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [CANDIDATE]: Term 1 election: Requested vote from peers 392a7502d1ce489881076810bd559790 (127.14.127.194:45549), e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937)
I20260923 20:21:41.734407 16905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d64c7f14e45e5af765bd5c27c858b" candidate_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790"
I20260923 20:21:41.734453 17040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5d64c7f14e45e5af765bd5c27c858b" candidate_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d11c10bf9e4d1395077f505fc30bf9"
I20260923 20:21:41.734602 16905 raft_consensus.cc:3037] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:41.734680 17040 raft_consensus.cc:3037] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:41.736588 16905 raft_consensus.cc:2454] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e85f99f835a42fdbaab20e9351a3cb3 in term 1.
I20260923 20:21:41.736608 17040 raft_consensus.cc:2454] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e85f99f835a42fdbaab20e9351a3cb3 in term 1.
I20260923 20:21:41.736949 16694 leader_election.cc:302] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, 392a7502d1ce489881076810bd559790; no voters:
I20260923 20:21:41.737390 17132 raft_consensus.cc:2781] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:41.737560 17132 raft_consensus.cc:686] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 1e85f99f835a42fdbaab20e9351a3cb3, State: Running, Role: LEADER
I20260923 20:21:41.737819 17132 consensus_queue.cc:237] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.740249 16603 catalog_manager.cc:5495] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } health_report { overall_health: UNKNOWN } } }
I20260923 20:21:41.769615 17210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:41.764738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.14.127.193:34177)
0923 20:21:41.765264 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:41.765268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:41.765286 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:21:41.765334 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:41.765337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:41.765394 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:41.765475 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:41.765756 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.766442 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.767931 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.768450 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.768552 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:41.768931 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:41.769044 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:41.769110 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:41.769209 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 20:21:41.769264 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":308,"threads_started":1}
I20260923 20:21:41.781690 17210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:41.777181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:37101 (local address 127.14.127.193:34177)
0923 20:21:41.777324 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:41.777328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:41.777744 (+ 416us) server_negotiation.cc:407] Connection header received
0923 20:21:41.777868 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:41.777872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:41.777927 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:41.777999 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:41.778371 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.779133 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.780351 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.780704 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.780791 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:41.781144 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:41.781229 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:41.781295 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:41.781410 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 20:21:41.781468 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260923 20:21:41.781821 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:41.777085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:34177 (local address 127.14.127.193:37101)
0923 20:21:41.777656 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:41.777675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:41.777776 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:41.778033 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:41.778036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:41.778048 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:21:41.778248 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:41.778254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.779248 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.779251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:41.780236 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:41.780247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.780833 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.780837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:41.780977 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:41.781018 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:21:41.781312 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:21:41.781319 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 20:21:41.781375 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 20:21:41.781467 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":314,"threads_started":1}
I20260923 20:21:41.789068 17040 raft_consensus.cc:1261] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Refusing update from remote peer 1e85f99f835a42fdbaab20e9351a3cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:41.789086 16905 raft_consensus.cc:1261] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Refusing update from remote peer 1e85f99f835a42fdbaab20e9351a3cb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:41.789575 17176 consensus_queue.cc:1035] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:41.789911 17132 consensus_queue.cc:1035] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:41.794718 17173 mvcc.cc:205] Tried to move back new op lower bound from 7332638317723414528 to 7332638317520183296. Current Snapshot: MvccSnapshot[applied={T|T < 7332638317723414528}]
I20260923 20:21:41.838539 16603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:41.838860 16603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:41.842095 16603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: Sending DeleteTablet for 3 replicas of tablet 2c5d64c7f14e45e5af765bd5c27c858b
I20260923 20:21:41.843065 16881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c5d64c7f14e45e5af765bd5c27c858b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43138
I20260923 20:21:41.843065 17020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c5d64c7f14e45e5af765bd5c27c858b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:33186
I20260923 20:21:41.843088 16741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c5d64c7f14e45e5af765bd5c27c858b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:60690
I20260923 20:21:41.843340 17191 tablet_replica.cc:330] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: stopping tablet replica
I20260923 20:21:41.843393 17192 tablet_replica.cc:330] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: stopping tablet replica
I20260923 20:21:41.843480 17191 raft_consensus.cc:2229] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:41.843617 17191 raft_consensus.cc:2258] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:41.843708 17192 raft_consensus.cc:2229] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:41.843868 17190 tablet_replica.cc:330] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: stopping tablet replica
I20260923 20:21:41.844023 17192 raft_consensus.cc:2258] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:41.844051 17190 raft_consensus.cc:2229] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:21:41.844532 17190 raft_consensus.cc:2258] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:41.845418 17192 ts_tablet_manager.cc:1905] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:41.845697 17190 ts_tablet_manager.cc:1905] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:41.845690 17191 ts_tablet_manager.cc:1905] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:41.851454 16603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:21:41.852188 16603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:21:41.854017 17192 ts_tablet_manager.cc:1918] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:41.854174 17192 log.cc:1196] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/2c5d64c7f14e45e5af765bd5c27c858b
I20260923 20:21:41.854462 17190 ts_tablet_manager.cc:1918] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:41.854696 17190 log.cc:1196] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/2c5d64c7f14e45e5af765bd5c27c858b
I20260923 20:21:41.854588 17192 ts_tablet_manager.cc:1939] T 2c5d64c7f14e45e5af765bd5c27c858b P e5d11c10bf9e4d1395077f505fc30bf9: Deleting consensus metadata
I20260923 20:21:41.855212 17190 ts_tablet_manager.cc:1939] T 2c5d64c7f14e45e5af765bd5c27c858b P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting consensus metadata
I20260923 20:21:41.856408 16589 catalog_manager.cc:4841] TS e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937): tablet 2c5d64c7f14e45e5af765bd5c27c858b (table test-table [id=d425457378754a89b43fbbbad1a00873]) successfully deleted
I20260923 20:21:41.856824 16589 catalog_manager.cc:4841] TS 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177): tablet 2c5d64c7f14e45e5af765bd5c27c858b (table test-table [id=d425457378754a89b43fbbbad1a00873]) successfully deleted
I20260923 20:21:41.859525 16881 tablet_service.cc:1467] Processing CreateTablet for tablet 5283450da7024500bc6dd2c5aadd8f76 (DEFAULT_TABLE table=test-table [id=941ab4e95530460d87d842a26be31bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:41.859764 17020 tablet_service.cc:1467] Processing CreateTablet for tablet 5283450da7024500bc6dd2c5aadd8f76 (DEFAULT_TABLE table=test-table [id=941ab4e95530460d87d842a26be31bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:41.859975 16881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5283450da7024500bc6dd2c5aadd8f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:41.859861 16741 tablet_service.cc:1467] Processing CreateTablet for tablet 5283450da7024500bc6dd2c5aadd8f76 (DEFAULT_TABLE table=test-table [id=941ab4e95530460d87d842a26be31bab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:21:41.860131 17020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5283450da7024500bc6dd2c5aadd8f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:41.860307 16741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5283450da7024500bc6dd2c5aadd8f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:41.867290 17203 tablet_bootstrap.cc:492] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: Bootstrap starting.
I20260923 20:21:41.868674 17191 ts_tablet_manager.cc:1918] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:21:41.868827 17191 log.cc:1196] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/2c5d64c7f14e45e5af765bd5c27c858b
I20260923 20:21:41.868942 17203 tablet_bootstrap.cc:654] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:41.870217 17201 tablet_bootstrap.cc:492] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: Bootstrap starting.
I20260923 20:21:41.870885 17202 tablet_bootstrap.cc:492] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: Bootstrap starting.
I20260923 20:21:41.872007 17201 tablet_bootstrap.cc:654] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:41.872824 17202 tablet_bootstrap.cc:654] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:41.875839 17203 tablet_bootstrap.cc:492] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: No bootstrap required, opened a new log
I20260923 20:21:41.876092 17203 ts_tablet_manager.cc:1397] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260923 20:21:41.877058 17203 raft_consensus.cc:348] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.877399 17203 raft_consensus.cc:374] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:41.877533 17203 raft_consensus.cc:729] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d11c10bf9e4d1395077f505fc30bf9, State: Initialized, Role: FOLLOWER
I20260923 20:21:41.877868 17203 consensus_queue.cc:260] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.878763 17203 ts_tablet_manager.cc:1428] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260923 20:21:41.878872 17191 ts_tablet_manager.cc:1939] T 2c5d64c7f14e45e5af765bd5c27c858b P 392a7502d1ce489881076810bd559790: Deleting consensus metadata
I20260923 20:21:41.880409 17201 tablet_bootstrap.cc:492] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: No bootstrap required, opened a new log
I20260923 20:21:41.880445 16591 catalog_manager.cc:4841] TS 392a7502d1ce489881076810bd559790 (127.14.127.194:45549): tablet 2c5d64c7f14e45e5af765bd5c27c858b (table test-table [id=d425457378754a89b43fbbbad1a00873]) successfully deleted
I20260923 20:21:41.880657 17201 ts_tablet_manager.cc:1397] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260923 20:21:41.881529 17201 raft_consensus.cc:348] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.881829 17201 raft_consensus.cc:374] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:41.881973 17201 raft_consensus.cc:729] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392a7502d1ce489881076810bd559790, State: Initialized, Role: FOLLOWER
I20260923 20:21:41.882301 17201 consensus_queue.cc:260] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.883000 17201 ts_tablet_manager.cc:1428] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:21:41.883133 17202 tablet_bootstrap.cc:492] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: No bootstrap required, opened a new log
I20260923 20:21:41.883461 17202 ts_tablet_manager.cc:1397] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260923 20:21:41.884298 17202 raft_consensus.cc:348] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.884630 17202 raft_consensus.cc:374] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:41.884775 17202 raft_consensus.cc:729] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e85f99f835a42fdbaab20e9351a3cb3, State: Initialized, Role: FOLLOWER
I20260923 20:21:41.885058 17202 consensus_queue.cc:260] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.886557 17202 ts_tablet_manager.cc:1428] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:21:41.903338 17115 raft_consensus.cc:482] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:21:41.903575 17115 raft_consensus.cc:504] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.904335 17115 leader_election.cc:288] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177), 392a7502d1ce489881076810bd559790 (127.14.127.194:45549)
I20260923 20:21:41.904879 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5283450da7024500bc6dd2c5aadd8f76" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" is_pre_election: true
I20260923 20:21:41.904992 16905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5283450da7024500bc6dd2c5aadd8f76" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790" is_pre_election: true
I20260923 20:21:41.905189 16767 raft_consensus.cc:2454] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 0.
I20260923 20:21:41.905230 16905 raft_consensus.cc:2454] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 0.
I20260923 20:21:41.905705 16971 leader_election.cc:302] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, e5d11c10bf9e4d1395077f505fc30bf9; no voters:
I20260923 20:21:41.906042 17115 raft_consensus.cc:2781] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:21:41.906142 17115 raft_consensus.cc:482] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:21:41.906291 17115 raft_consensus.cc:3037] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:41.908366 17115 raft_consensus.cc:504] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.908947 17115 leader_election.cc:288] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 election: Requested vote from peers 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177), 392a7502d1ce489881076810bd559790 (127.14.127.194:45549)
I20260923 20:21:41.909360 16905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5283450da7024500bc6dd2c5aadd8f76" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392a7502d1ce489881076810bd559790"
I20260923 20:21:41.909387 16767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5283450da7024500bc6dd2c5aadd8f76" candidate_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e85f99f835a42fdbaab20e9351a3cb3"
I20260923 20:21:41.909585 16767 raft_consensus.cc:3037] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:41.909598 16905 raft_consensus.cc:3037] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:41.912580 16905 raft_consensus.cc:2454] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 1.
I20260923 20:21:41.912648 16767 raft_consensus.cc:2454] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d11c10bf9e4d1395077f505fc30bf9 in term 1.
I20260923 20:21:41.913096 16973 leader_election.cc:302] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1e85f99f835a42fdbaab20e9351a3cb3, 392a7502d1ce489881076810bd559790, e5d11c10bf9e4d1395077f505fc30bf9; no voters:
I20260923 20:21:41.913424 17115 raft_consensus.cc:2781] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:41.913636 17115 raft_consensus.cc:686] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 LEADER]: Becoming Leader. State: Replica: e5d11c10bf9e4d1395077f505fc30bf9, State: Running, Role: LEADER
I20260923 20:21:41.913992 17115 consensus_queue.cc:237] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } }
I20260923 20:21:41.916689 16603 catalog_manager.cc:5495] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d11c10bf9e4d1395077f505fc30bf9" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 33937 } health_report { overall_health: HEALTHY } } }
I20260923 20:21:41.943539 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:41.938121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.14.127.195:33937)
0923 20:21:41.938913 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:41.938919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:41.938942 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:21:41.939002 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:41.939007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:41.939083 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:41.939189 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:41.939578 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.940345 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.941664 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:41.942136 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:41.942271 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:41.942753 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:41.942859 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:41.942959 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:41.943113 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 20:21:41.943186 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":548,"threads_started":1}
I20260923 20:21:41.952768 16767 raft_consensus.cc:1261] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Refusing update from remote peer e5d11c10bf9e4d1395077f505fc30bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:41.952773 16905 raft_consensus.cc:1261] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Refusing update from remote peer e5d11c10bf9e4d1395077f505fc30bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:41.953363 17185 consensus_queue.cc:1035] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "392a7502d1ce489881076810bd559790" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:41.953707 17115 consensus_queue.cc:1035] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e85f99f835a42fdbaab20e9351a3cb3" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:21:41.998032 17172 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 20:21:43.602883 16603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:43.603133 16603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:21:43.607283 16603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad54b7dfca3540ac92a188ac7ab28ba5: Sending DeleteTablet for 3 replicas of tablet 5283450da7024500bc6dd2c5aadd8f76
I20260923 20:21:43.608176 16881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5283450da7024500bc6dd2c5aadd8f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:43 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43138
I20260923 20:21:43.608161 17020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5283450da7024500bc6dd2c5aadd8f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:43 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:33186
I20260923 20:21:43.608582 16742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5283450da7024500bc6dd2c5aadd8f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:21:43 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:60690
I20260923 20:21:43.608800 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16676
I20260923 20:21:43.608852 17249 tablet_replica.cc:330] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: stopping tablet replica
I20260923 20:21:43.609102 17249 raft_consensus.cc:2229] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:21:43.609684 17249 raft_consensus.cc:2258] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:43.611006 17249 ts_tablet_manager.cc:1905] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:43.619556 17251 tablet_replica.cc:330] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: stopping tablet replica
I20260923 20:21:43.620016 17251 raft_consensus.cc:2229] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:43.620384 17251 raft_consensus.cc:2258] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:43.620406 17249 ts_tablet_manager.cc:1918] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:21:43.621132 17249 log.cc:1196] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/5283450da7024500bc6dd2c5aadd8f76
I20260923 20:21:43.621773 17249 ts_tablet_manager.cc:1939] T 5283450da7024500bc6dd2c5aadd8f76 P e5d11c10bf9e4d1395077f505fc30bf9: Deleting consensus metadata
I20260923 20:21:43.621966 17251 ts_tablet_manager.cc:1905] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:43.623406 16589 catalog_manager.cc:4841] TS e5d11c10bf9e4d1395077f505fc30bf9 (127.14.127.195:33937): tablet 5283450da7024500bc6dd2c5aadd8f76 (table test-table [id=941ab4e95530460d87d842a26be31bab]) successfully deleted
I20260923 20:21:43.623885 17250 tablet_replica.cc:330] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: stopping tablet replica
I20260923 20:21:43.624125 17250 raft_consensus.cc:2229] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:21:43.624365 17250 raft_consensus.cc:2258] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:21:43.626209 17250 ts_tablet_manager.cc:1905] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:21:43.630399 17251 ts_tablet_manager.cc:1918] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:21:43.630602 17251 log.cc:1196] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/5283450da7024500bc6dd2c5aadd8f76
I20260923 20:21:43.631084 17251 ts_tablet_manager.cc:1939] T 5283450da7024500bc6dd2c5aadd8f76 P 1e85f99f835a42fdbaab20e9351a3cb3: Deleting consensus metadata
I20260923 20:21:43.632090 16589 catalog_manager.cc:4841] TS 1e85f99f835a42fdbaab20e9351a3cb3 (127.14.127.193:34177): tablet 5283450da7024500bc6dd2c5aadd8f76 (table test-table [id=941ab4e95530460d87d842a26be31bab]) successfully deleted
I20260923 20:21:43.633188 17250 ts_tablet_manager.cc:1918] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:21:43.633296 17250 log.cc:1196] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/5283450da7024500bc6dd2c5aadd8f76
I20260923 20:21:43.633738 17250 ts_tablet_manager.cc:1939] T 5283450da7024500bc6dd2c5aadd8f76 P 392a7502d1ce489881076810bd559790: Deleting consensus metadata
I20260923 20:21:43.634967 16591 catalog_manager.cc:4841] TS 392a7502d1ce489881076810bd559790 (127.14.127.194:45549): tablet 5283450da7024500bc6dd2c5aadd8f76 (table test-table [id=941ab4e95530460d87d842a26be31bab]) successfully deleted
Sep 23 20:21:43 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:21:43.642266 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:43.610700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.14.127.193:34177)
0923 20:21:43.621108 (+ 10408us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:43.621117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:43.625362 (+ 4245us) server_negotiation.cc:407] Connection header received
0923 20:21:43.625417 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:43.625421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:43.625482 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:43.625560 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:43.627373 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:43.628812 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:43.630785 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:43.631589 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:43.631753 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:43.636674 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:43.636690 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:43.636706 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:43.636751 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:43.639516 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:43.640202 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:43.640210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:43.640212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:43.640340 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:43.640742 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:43.640746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:43.640748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:43.641090 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:43.641402 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:43.641667 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 20:21:43.641927 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10216,"thread_start_us":10103,"threads_started":1}
W20260923 20:21:43.642990 16807 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55020
W20260923 20:21:43.643643 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:43.643826 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 16676
I20260923 20:21:43.668879 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16816
Sep 23 20:21:43 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:21:43.683645 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:43.669626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.14.127.194:45549)
0923 20:21:43.670164 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:43.670168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:43.670184 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:21:43.670236 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:43.670240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:43.670295 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:43.670384 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:43.671647 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:43.672419 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:43.673670 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:43.674050 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:43.674141 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:43.678360 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:43.678372 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:43.678387 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:43.678424 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:43.681042 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:43.681663 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:43.681670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:43.681672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:43.681830 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:43.682141 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:43.682144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:43.682146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:43.682470 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:43.682772 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:43.683169 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 20:21:43.683373 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":314,"threads_started":1}
W20260923 20:21:43.684335 16946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43166
W20260923 20:21:43.684945 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:43.685065 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 16816
I20260923 20:21:43.703301 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16955
Sep 23 20:21:43 dist-test-slave-wmd0 krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:21:43.718062 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:43.703976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.14.127.195:33937)
0923 20:21:43.704451 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:43.704456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:43.704470 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:21:43.704525 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:43.704529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:43.704589 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:43.704707 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:43.705974 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:43.706768 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:43.707983 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:43.708420 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:43.708581 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:43.712787 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:43.712801 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:43.712817 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:43.712853 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:43.715671 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:43.716307 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:43.716313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:43.716315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:43.716437 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:43.716757 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:43.716760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:43.716761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:43.717086 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:43.717316 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:43.717544 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 20:21:43.717699 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":230,"threads_started":1}
W20260923 20:21:43.718822 17085 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37248
W20260923 20:21:43.719383 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16955: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:43.719561 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 16955
I20260923 20:21:43.741160 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 16573
I20260923 20:21:43.742409 16645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:43.857474 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 16573
2026-09-23T20:21:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8217 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:43 dist-test-slave-wmd0 krb5kdc[17259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:43 dist-test-slave-wmd0 krb5kdc[17259](info): set up 2 sockets
Sep 23 20:21:43 dist-test-slave-wmd0 krb5kdc[17259](info): commencing operation
krb5kdc: starting...
W20260923 20:21:44.943029 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:21:45 dist-test-slave-wmd0 krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:45.130602 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:35233
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:35233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:45.445225 17272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:45.445572 17272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:45.445746 17272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:45.453197 17272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:45.453369 17272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:45.453481 17272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:45.453605 17272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:45.453691 17272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:45.463423 17272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:35233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:35233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:45.464751 17272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:45.466065 17272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:45.478571 17277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:45.478662 17280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:45.478710 17278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:45.479123 17272 server_base.cc:1034] running on GCE node
I20260923 20:21:45.480069 17272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:45.481436 17272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:45.482707 17272 hybrid_clock.cc:648] HybridClock initialized: now 1790194905482656 us; error 33 us; skew 500 ppm
I20260923 20:21:45.483076 17272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:45 dist-test-slave-wmd0 krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:45.486238 17272 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:45.487833 17272 webserver.cc:460] Webserver started at http://127.14.127.254:38717/ using document root <none> and password file <none>
I20260923 20:21:45.488529 17272 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:45.488680 17272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:45.488991 17272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:45.491995 17272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "a8ebba619710455cb5fefce2df07d005"
format_stamp: "Formatted at 2026-09-23 20:21:45 on dist-test-slave-wmd0"
server_key: "ca05d548a2353712ea972b2d407d311a"
server_key_iv: "84865390fc6b5329e8ef1bee05198d10"
server_key_version: "encryptionkey@0"
I20260923 20:21:45.492764 17272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "a8ebba619710455cb5fefce2df07d005"
format_stamp: "Formatted at 2026-09-23 20:21:45 on dist-test-slave-wmd0"
server_key: "ca05d548a2353712ea972b2d407d311a"
server_key_iv: "84865390fc6b5329e8ef1bee05198d10"
server_key_version: "encryptionkey@0"
I20260923 20:21:45.498171 17272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:21:45.501863 17287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:45.503407 17272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:45.503611 17272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "a8ebba619710455cb5fefce2df07d005"
format_stamp: "Formatted at 2026-09-23 20:21:45 on dist-test-slave-wmd0"
server_key: "ca05d548a2353712ea972b2d407d311a"
server_key_iv: "84865390fc6b5329e8ef1bee05198d10"
server_key_version: "encryptionkey@0"
I20260923 20:21:45.503827 17272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:45.519377 17272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:45.523433 17272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:45.523725 17272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:45.542055 17272 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:35233
I20260923 20:21:45.542088 17338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:35233 every 8 connection(s)
I20260923 20:21:45.543475 17272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:45.547305 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:45.547479 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 17272
I20260923 20:21:45.547724 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:45.548488 14847 external_mini_cluster.cc:1428] Setting key e02fff62881f1d38c0bd01076a571b30
I20260923 20:21:45.561535 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005: Bootstrap starting.
Sep 23 20:21:45 dist-test-slave-wmd0 krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:45.565171 17339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:45.566424 17339 log.cc:824] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:45.569351 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005: No bootstrap required, opened a new log
I20260923 20:21:45.572084 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:45.550524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.14.127.254:35233)
0923 20:21:45.551396 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:45.551407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:45.551444 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:21:45.553771 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:45.553795 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:45.554245 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:45.554692 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:45.556362 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:45.557659 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:45.559493 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:45.560156 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:45.560287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:45.564810 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:45.564818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:45.564833 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:45.564868 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:45.568078 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:45.568747 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:45.568753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:45.568759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:45.568926 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:45.569322 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:45.569325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:45.569327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:45.569911 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:45.570242 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:45.570682 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:21:45.571105 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":404,"threads_started":1}
I20260923 20:21:45.577889 17339 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ebba619710455cb5fefce2df07d005" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35233 } }
I20260923 20:21:45.578351 17339 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:45.578485 17339 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ebba619710455cb5fefce2df07d005, State: Initialized, Role: FOLLOWER
I20260923 20:21:45.579025 17339 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ebba619710455cb5fefce2df07d005" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35233 } }
I20260923 20:21:45.579269 17339 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:45.579391 17339 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:45.579571 17339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:45.581652 17339 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ebba619710455cb5fefce2df07d005" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35233 } }
I20260923 20:21:45.582142 17339 leader_election.cc:302] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8ebba619710455cb5fefce2df07d005; no voters:
I20260923 20:21:45.582777 17339 leader_election.cc:288] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:45.582916 17344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:45.583572 17344 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [term 1 LEADER]: Becoming Leader. State: Replica: a8ebba619710455cb5fefce2df07d005, State: Running, Role: LEADER
I20260923 20:21:45.584120 17344 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ebba619710455cb5fefce2df07d005" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35233 } }
I20260923 20:21:45.584815 17339 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:45.587045 17346 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8ebba619710455cb5fefce2df07d005. Latest consensus state: current_term: 1 leader_uuid: "a8ebba619710455cb5fefce2df07d005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ebba619710455cb5fefce2df07d005" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35233 } } }
I20260923 20:21:45.587458 17346 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:45.587968 17345 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8ebba619710455cb5fefce2df07d005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ebba619710455cb5fefce2df07d005" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35233 } } }
I20260923 20:21:45.588255 17345 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:45.588989 17353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:45.593851 17353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:45.602581 17353 catalog_manager.cc:1347] Generated new cluster ID: aaa669c5fae940eea90e8ad20cb81df1
I20260923 20:21:45.602706 17353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:45.632285 17353 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:45.634256 17353 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:45.646162 17353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8ebba619710455cb5fefce2df07d005: Generated new TSK 0
I20260923 20:21:45.647280 17353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:21:45.714561 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35233
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:46.173625 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:46.173907 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:46.174075 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:46.181313 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:46.181422 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:46.181635 17365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:21:46.191347 17365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:46.192657 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:46.193948 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:46.206893 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:46.206887 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:46.207466 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:46.207598 17365 server_base.cc:1034] running on GCE node
I20260923 20:21:46.208457 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:46.209455 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:46.210706 17365 hybrid_clock.cc:648] HybridClock initialized: now 1790194906210654 us; error 46 us; skew 500 ppm
I20260923 20:21:46.211024 17365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:46 dist-test-slave-wmd0 krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:46.214145 17365 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:21:46.215598 17365 webserver.cc:460] Webserver started at http://127.14.127.193:42277/ using document root <none> and password file <none>
I20260923 20:21:46.216276 17365 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:46.216419 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:46.216729 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:46.219653 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "c047692f03b54ac7b72b5c002dc8382f"
format_stamp: "Formatted at 2026-09-23 20:21:46 on dist-test-slave-wmd0"
server_key: "4c5cfe924cc05146a2a2f5829602bcc0"
server_key_iv: "37c46a11bb4ee6b909774681665c15d7"
server_key_version: "encryptionkey@0"
I20260923 20:21:46.220430 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "c047692f03b54ac7b72b5c002dc8382f"
format_stamp: "Formatted at 2026-09-23 20:21:46 on dist-test-slave-wmd0"
server_key: "4c5cfe924cc05146a2a2f5829602bcc0"
server_key_iv: "37c46a11bb4ee6b909774681665c15d7"
server_key_version: "encryptionkey@0"
I20260923 20:21:46.225409 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:21:46.228739 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:46.229962 17365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 20:21:46.230132 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "c047692f03b54ac7b72b5c002dc8382f"
format_stamp: "Formatted at 2026-09-23 20:21:46 on dist-test-slave-wmd0"
server_key: "4c5cfe924cc05146a2a2f5829602bcc0"
server_key_iv: "37c46a11bb4ee6b909774681665c15d7"
server_key_version: "encryptionkey@0"
I20260923 20:21:46.230342 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:46.251812 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:46.256232 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:46.256536 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:46.257496 17365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:46.258867 17365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:46.258976 17365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:46.259135 17365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:46.259208 17365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:46.292523 17365 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:41839
I20260923 20:21:46.292568 17492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:41839 every 8 connection(s)
I20260923 20:21:46.293831 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:21:46.302567 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 17365
I20260923 20:21:46.302822 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:21:46.303431 14847 external_mini_cluster.cc:1428] Setting key 6676d4b866ea7b6c8888dfa8bc2896ea
Sep 23 20:21:46 dist-test-slave-wmd0 krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:46.315793 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:46.296403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:38999 (local address 127.14.127.254:35233)
0923 20:21:46.296877 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:46.296883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:46.297793 (+ 910us) server_negotiation.cc:407] Connection header received
0923 20:21:46.299661 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:46.299666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:46.299733 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:46.299830 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:46.301393 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:46.302280 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:46.304074 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:46.304707 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:46.304834 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:46.309090 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:46.309095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:46.309098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:46.309130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:46.312356 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:46.313084 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:46.313091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:46.313098 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:46.313246 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:46.313667 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:46.313674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:46.313680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:46.314076 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:46.314258 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:46.315013 (+ 755us) server_negotiation.cc:299] Negotiation successful
0923 20:21:46.315284 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":218,"threads_started":1}
I20260923 20:21:46.316373 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:46.296921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35233 (local address 127.14.127.193:38999)
0923 20:21:46.297631 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:46.297673 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:46.299404 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:46.300032 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:46.300040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:46.300621 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:46.301198 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:46.301216 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:46.302450 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:46.302454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:46.303910 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:46.303922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:46.304950 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:46.304959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:46.305284 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:46.305983 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:46.306005 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:46.308868 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:46.312490 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:46.312497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:46.312510 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:46.312925 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:46.313350 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:46.313354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:46.313356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:46.313542 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:46.314205 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:46.314229 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:46.314549 (+ 320us) client_negotiation.cc:769] Sending connection context
0923 20:21:46.314852 (+ 303us) client_negotiation.cc:240] Negotiation successful
0923 20:21:46.315230 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":265,"threads_started":1}
I20260923 20:21:46.318197 17493 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35233
I20260923 20:21:46.318570 17493 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:46.319542 17493 heartbeater.cc:511] Master 127.14.127.254:35233 requested a full tablet report, sending...
I20260923 20:21:46.321743 17304 ts_manager.cc:194] Registered new tserver with Master: c047692f03b54ac7b72b5c002dc8382f (127.14.127.193:41839)
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:21:46.323380 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:38999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:21:46.345902 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35233
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:46.669363 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:46.669763 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:46.669970 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:46.677573 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:46.677739 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:46.677911 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:21:46.688050 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:46.689347 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:46.690749 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:46.703728 17504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:46.703775 17507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:46.703801 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:46.704674 17499 server_base.cc:1034] running on GCE node
I20260923 20:21:46.705267 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:46.706152 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:46.707351 17499 hybrid_clock.cc:648] HybridClock initialized: now 1790194906707287 us; error 51 us; skew 500 ppm
I20260923 20:21:46.707666 17499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:46 dist-test-slave-wmd0 krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:46.721065 17499 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:21:46.722729 17499 webserver.cc:460] Webserver started at http://127.14.127.194:39309/ using document root <none> and password file <none>
I20260923 20:21:46.723441 17499 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:46.723582 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:46.723902 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:46.726848 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "8485ca1b1cb448b280c36f71393c3749"
format_stamp: "Formatted at 2026-09-23 20:21:46 on dist-test-slave-wmd0"
server_key: "6638c23c2619e6093a282c9569474f3c"
server_key_iv: "9a971ad2db201fb7f2c249f3ba6b4302"
server_key_version: "encryptionkey@0"
I20260923 20:21:46.727648 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "8485ca1b1cb448b280c36f71393c3749"
format_stamp: "Formatted at 2026-09-23 20:21:46 on dist-test-slave-wmd0"
server_key: "6638c23c2619e6093a282c9569474f3c"
server_key_iv: "9a971ad2db201fb7f2c249f3ba6b4302"
server_key_version: "encryptionkey@0"
I20260923 20:21:46.732777 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:21:46.736529 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:46.737797 17499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:21:46.737968 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "8485ca1b1cb448b280c36f71393c3749"
format_stamp: "Formatted at 2026-09-23 20:21:46 on dist-test-slave-wmd0"
server_key: "6638c23c2619e6093a282c9569474f3c"
server_key_iv: "9a971ad2db201fb7f2c249f3ba6b4302"
server_key_version: "encryptionkey@0"
I20260923 20:21:46.738183 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:46.751771 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:46.756541 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:46.756867 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:46.757898 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:46.759342 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:46.759416 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:46.759572 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:46.759626 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:46.793272 17499 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:46743
I20260923 20:21:46.793320 17626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:46743 every 8 connection(s)
I20260923 20:21:46.794667 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:21:46.804183 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 17499
I20260923 20:21:46.804392 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:21:46.804792 14847 external_mini_cluster.cc:1428] Setting key 4c12e8160c33cc23100206bf436d6516
Sep 23 20:21:46 dist-test-slave-wmd0 krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:46.814502 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:46.797252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:45517 (local address 127.14.127.254:35233)
0923 20:21:46.797434 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:46.797438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:46.798540 (+ 1102us) server_negotiation.cc:407] Connection header received
0923 20:21:46.800372 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:46.800377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:46.800472 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:46.800593 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:46.802101 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:46.802968 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:46.804665 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:46.805181 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:46.805315 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:46.808839 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:46.808844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:46.808847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:46.808880 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:46.811488 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:46.812241 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:46.812247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:46.812251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:46.812387 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:46.812826 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:46.812831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:46.812835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:46.813099 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:46.813251 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:46.813936 (+ 685us) server_negotiation.cc:299] Negotiation successful
0923 20:21:46.814168 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:21:46.815538 17629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:46.797662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35233 (local address 127.14.127.194:45517)
0923 20:21:46.798371 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:46.798415 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:46.800095 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:46.800835 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:46.800844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:46.801371 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:46.801925 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:46.801941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:46.803117 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:46.803122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:46.804523 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:46.804538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:46.805416 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:46.805423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:46.805658 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:46.806338 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:46.806360 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:46.808614 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:46.811660 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:46.811669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:46.811687 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:46.812121 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:46.812503 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:46.812508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:46.812512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:46.812709 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:46.813227 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:46.813234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:46.813591 (+ 357us) client_negotiation.cc:769] Sending connection context
0923 20:21:46.813856 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 20:21:46.814402 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":275,"threads_started":1}
I20260923 20:21:46.817098 17627 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35233
I20260923 20:21:46.817435 17627 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:46.818223 17627 heartbeater.cc:511] Master 127.14.127.254:35233 requested a full tablet report, sending...
I20260923 20:21:46.819774 17304 ts_manager.cc:194] Registered new tserver with Master: 8485ca1b1cb448b280c36f71393c3749 (127.14.127.194:46743)
I20260923 20:21:46.820590 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:45517
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:21:46.845855 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35233
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:47.158953 17632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:47.159242 17632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:47.159413 17632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:47.166531 17632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:47.166656 17632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:47.166853 17632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:21:47.176472 17632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:47.177762 17632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:47.179112 17632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:47.192190 17638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:47.192301 17637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:47.192212 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:47.201094 17632 server_base.cc:1034] running on GCE node
I20260923 20:21:47.201668 17632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:47.202574 17632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:47.203836 17632 hybrid_clock.cc:648] HybridClock initialized: now 1790194907203770 us; error 62 us; skew 500 ppm
I20260923 20:21:47.204205 17632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:47 dist-test-slave-wmd0 krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194907, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:47.207311 17632 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:21:47.208984 17632 webserver.cc:460] Webserver started at http://127.14.127.195:46179/ using document root <none> and password file <none>
I20260923 20:21:47.209714 17632 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:47.209846 17632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:47.210170 17632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:47.213146 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "9dde46c921004b6cafab60f84c711ca0"
format_stamp: "Formatted at 2026-09-23 20:21:47 on dist-test-slave-wmd0"
server_key: "8d8fee55b20aecaf7233fca0b831ebd6"
server_key_iv: "eab2d2caedb959b8df6d29e9c71f507c"
server_key_version: "encryptionkey@0"
I20260923 20:21:47.213932 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "9dde46c921004b6cafab60f84c711ca0"
format_stamp: "Formatted at 2026-09-23 20:21:47 on dist-test-slave-wmd0"
server_key: "8d8fee55b20aecaf7233fca0b831ebd6"
server_key_iv: "eab2d2caedb959b8df6d29e9c71f507c"
server_key_version: "encryptionkey@0"
I20260923 20:21:47.219178 17632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 20:21:47.222535 17647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:47.223881 17632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:47.224059 17632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "9dde46c921004b6cafab60f84c711ca0"
format_stamp: "Formatted at 2026-09-23 20:21:47 on dist-test-slave-wmd0"
server_key: "8d8fee55b20aecaf7233fca0b831ebd6"
server_key_iv: "eab2d2caedb959b8df6d29e9c71f507c"
server_key_version: "encryptionkey@0"
I20260923 20:21:47.224260 17632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:47.247484 17632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:47.251307 17632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:47.251602 17632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:47.252584 17632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:47.253849 17632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:47.253952 17632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:47.254097 17632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:47.254169 17632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:47.286877 17632 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:36193
I20260923 20:21:47.286938 17759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:36193 every 8 connection(s)
I20260923 20:21:47.288197 17632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:21:47.293373 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 17632
I20260923 20:21:47.293579 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:21:47.294088 14847 external_mini_cluster.cc:1428] Setting key a7a5c47f9820c6855819d68a921bc1fc
Sep 23 20:21:47 dist-test-slave-wmd0 krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194907, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:47.309439 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:47.290743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:58831 (local address 127.14.127.254:35233)
0923 20:21:47.290922 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:47.290926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:47.292191 (+ 1265us) server_negotiation.cc:407] Connection header received
0923 20:21:47.294591 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:47.294597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:47.294726 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:47.294835 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:47.296643 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:47.297601 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:47.299210 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:47.299602 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:47.299734 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:47.303044 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:47.303049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:47.303052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:47.303086 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:47.305843 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:47.306569 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:47.306574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:47.306576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:47.306745 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:47.307284 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:47.307288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:47.307290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:47.307750 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:47.307901 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:47.308866 (+ 965us) server_negotiation.cc:299] Negotiation successful
0923 20:21:47.309095 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:21:47.310460 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:47.291230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35233 (local address 127.14.127.195:58831)
0923 20:21:47.292026 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:47.292082 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:47.294249 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:47.295148 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:47.295156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:47.295762 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:47.296474 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:47.296493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:47.297748 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:47.297753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:47.299052 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:47.299071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:47.299817 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:47.299825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:47.300055 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:47.300721 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:47.300743 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:47.302832 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:47.305988 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:47.305996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:47.306012 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:47.306427 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:47.306856 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:47.306861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:47.306864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:47.307147 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:47.307902 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:47.307910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:47.308367 (+ 457us) client_negotiation.cc:769] Sending connection context
0923 20:21:47.308707 (+ 340us) client_negotiation.cc:240] Negotiation successful
0923 20:21:47.309229 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":263,"threads_started":1}
I20260923 20:21:47.311759 17760 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35233
I20260923 20:21:47.312067 17760 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:47.312760 17760 heartbeater.cc:511] Master 127.14.127.254:35233 requested a full tablet report, sending...
I20260923 20:21:47.314466 17304 ts_manager.cc:194] Registered new tserver with Master: 9dde46c921004b6cafab60f84c711ca0 (127.14.127.195:36193)
I20260923 20:21:47.315305 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:58831
I20260923 20:21:47.321762 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:21:47.326507 17493 heartbeater.cc:503] Master 127.14.127.254:35233 was elected leader, sending a full tablet report...
W20260923 20:21:47.351256 17495 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:47.348765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.14.127.254:35233)
0923 20:21:47.348949 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:47.348954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:47.349444 (+ 490us) server_negotiation.cc:407] Connection header received
0923 20:21:47.349623 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:47.349627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:47.349691 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:47.349782 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:47.350980 (+ 1198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34144: BlockingRecv error: recv got EOF from 127.0.0.1:34144 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:21:47.355088 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17365
W20260923 20:21:47.357772 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17365: Not authorized: Client connection negotiation failed: client connection to 127.14.127.193:41839: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:21:47.357963 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 17365
I20260923 20:21:47.373410 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17499
W20260923 20:21:47.376081 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17499: Not authorized: Client connection negotiation failed: client connection to 127.14.127.194:46743: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:21:47.376245 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 17499
I20260923 20:21:47.391769 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17632
W20260923 20:21:47.394711 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17632: Not authorized: Client connection negotiation failed: client connection to 127.14.127.195:36193: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:21:47.394835 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 17632
I20260923 20:21:47.410265 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17272
I20260923 20:21:47.411535 17334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:47.514704 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 17272
2026-09-23T20:21:47Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3655 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:47 dist-test-slave-wmd0 krb5kdc[17777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:47 dist-test-slave-wmd0 krb5kdc[17777](info): set up 2 sockets
Sep 23 20:21:47 dist-test-slave-wmd0 krb5kdc[17777](info): commencing operation
krb5kdc: starting...
W20260923 20:21:48.692570 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260923 20:21:49.108404 14847 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.416s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260923 20:21:49.338258 14847 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.230s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260923 20:21:49.571194 14847 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.233s user 0.004s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/test-user.keytab.
W20260923 20:21:49.731451 14847 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.160s user 0.000s sys 0.004s
Sep 23 20:21:49 dist-test-slave-wmd0 krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:49.814538 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:38743
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:38743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:50.133461 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:50.133754 17790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:50.133843 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:50.140851 17790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:50.140951 17790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:50.140997 17790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:50.141054 17790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:50.141103 17790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:50.150583 17790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:38743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:50.151810 17790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:50.153021 17790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:50.164920 17795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:50.164839 17798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:50.164891 17796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:50.166100 17790 server_base.cc:1034] running on GCE node
I20260923 20:21:50.166952 17790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:50.168329 17790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:50.169567 17790 hybrid_clock.cc:648] HybridClock initialized: now 1790194910169500 us; error 51 us; skew 500 ppm
I20260923 20:21:50.169910 17790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:50 dist-test-slave-wmd0 krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194910, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:50.172916 17790 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:50.174365 17790 webserver.cc:460] Webserver started at http://127.14.127.254:46593/ using document root <none> and password file <none>
I20260923 20:21:50.175109 17790 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:50.175253 17790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:50.175562 17790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:50.178462 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "aaedb140db6f4ee88a7238aeab6c3b3e"
format_stamp: "Formatted at 2026-09-23 20:21:50 on dist-test-slave-wmd0"
server_key: "85086d3f8eaec25a8027cfcfa93cddd7"
server_key_iv: "1c3437ac025fb2e4d8dea28cbca876d5"
server_key_version: "encryptionkey@0"
I20260923 20:21:50.179340 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "aaedb140db6f4ee88a7238aeab6c3b3e"
format_stamp: "Formatted at 2026-09-23 20:21:50 on dist-test-slave-wmd0"
server_key: "85086d3f8eaec25a8027cfcfa93cddd7"
server_key_iv: "1c3437ac025fb2e4d8dea28cbca876d5"
server_key_version: "encryptionkey@0"
I20260923 20:21:50.184226 17790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:21:50.187791 17805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:50.189103 17790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:21:50.189311 17790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "aaedb140db6f4ee88a7238aeab6c3b3e"
format_stamp: "Formatted at 2026-09-23 20:21:50 on dist-test-slave-wmd0"
server_key: "85086d3f8eaec25a8027cfcfa93cddd7"
server_key_iv: "1c3437ac025fb2e4d8dea28cbca876d5"
server_key_version: "encryptionkey@0"
I20260923 20:21:50.189527 17790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:50.203503 17790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:50.207489 17790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:50.207777 17790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:50.226315 17790 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:38743
I20260923 20:21:50.226354 17856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:38743 every 8 connection(s)
I20260923 20:21:50.227757 17790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:50.231370 17857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:50.231735 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 17790
I20260923 20:21:50.231947 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:50.232585 14847 external_mini_cluster.cc:1428] Setting key af224715a484e870aa0de5e58316f7fd
I20260923 20:21:50.244652 17857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e: Bootstrap starting.
Sep 23 20:21:50 dist-test-slave-wmd0 krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:50.248339 17857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:50.249696 17857 log.cc:824] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:50.252974 17857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e: No bootstrap required, opened a new log
I20260923 20:21:50.256520 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:50.234319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.14.127.254:38743)
0923 20:21:50.235115 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:50.235129 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:50.235167 (+ 38us) server_negotiation.cc:407] Connection header received
0923 20:21:50.237174 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:50.237195 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:50.237589 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:50.237915 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:50.239795 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:50.241067 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:50.242852 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:50.243477 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:50.243613 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:50.248118 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:50.248131 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:50.248151 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:50.248208 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:50.252015 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:50.252695 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:50.252702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:50.252710 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:50.252901 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:50.253305 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:50.253310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:50.253313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:50.254048 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:50.254347 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:50.254664 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 20:21:50.255115 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260923 20:21:50.260370 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:21:50.262297 17857 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaedb140db6f4ee88a7238aeab6c3b3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38743 } }
I20260923 20:21:50.262653 17857 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:50.262756 17857 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaedb140db6f4ee88a7238aeab6c3b3e, State: Initialized, Role: FOLLOWER
I20260923 20:21:50.263383 17857 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaedb140db6f4ee88a7238aeab6c3b3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38743 } }
I20260923 20:21:50.263607 17857 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:50.263732 17857 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:50.263898 17857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:50.265996 17857 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaedb140db6f4ee88a7238aeab6c3b3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38743 } }
I20260923 20:21:50.266497 17857 leader_election.cc:302] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaedb140db6f4ee88a7238aeab6c3b3e; no voters:
I20260923 20:21:50.267165 17857 leader_election.cc:288] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:50.267274 17863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:50.268616 17863 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [term 1 LEADER]: Becoming Leader. State: Replica: aaedb140db6f4ee88a7238aeab6c3b3e, State: Running, Role: LEADER
I20260923 20:21:50.269167 17863 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaedb140db6f4ee88a7238aeab6c3b3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38743 } }
I20260923 20:21:50.269584 17857 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:50.273805 17865 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaedb140db6f4ee88a7238aeab6c3b3e. Latest consensus state: current_term: 1 leader_uuid: "aaedb140db6f4ee88a7238aeab6c3b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaedb140db6f4ee88a7238aeab6c3b3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38743 } } }
I20260923 20:21:50.274190 17865 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:50.281934 17864 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaedb140db6f4ee88a7238aeab6c3b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaedb140db6f4ee88a7238aeab6c3b3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38743 } } }
I20260923 20:21:50.282300 17864 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:50.282508 17869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:50.289388 17869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:50.301957 17869 catalog_manager.cc:1347] Generated new cluster ID: 0c9c51aedf774193b6ae01d65459508f
I20260923 20:21:50.302078 17869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:50.316972 17869 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:50.318661 17869 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:50.320111 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:50.310321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.14.127.254:38743)
0923 20:21:50.310492 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:50.310504 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:50.311269 (+ 765us) server_negotiation.cc:407] Connection header received
0923 20:21:50.311460 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:50.311466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:50.311547 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:50.311649 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:50.312912 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:50.314053 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:50.315929 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:50.316754 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:50.316885 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:50.318179 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:50.318186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:50.318191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:21:50.319210 (+ 1019us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:50.319284 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:21:50.319313 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:50.319433 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:50.319602 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 20:21:50.319749 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":56}
I20260923 20:21:50.327034 17869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaedb140db6f4ee88a7238aeab6c3b3e: Generated new TSK 0
I20260923 20:21:50.327819 17869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:21:50.337419 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17790
I20260923 20:21:50.338641 17852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:50.426671 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 17790
2026-09-23T20:21:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2905 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:50 dist-test-slave-wmd0 krb5kdc[17891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:50 dist-test-slave-wmd0 krb5kdc[17891](info): set up 2 sockets
Sep 23 20:21:50 dist-test-slave-wmd0 krb5kdc[17891](info): commencing operation
krb5kdc: starting...
W20260923 20:21:51.503139 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:21:51 dist-test-slave-wmd0 krb5kdc[17891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:51.691818 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:41899
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:41899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:52.006079 17904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:52.006438 17904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:52.006655 17904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:52.013855 17904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:52.013996 17904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:52.014091 17904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:52.014178 17904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:52.014246 17904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:52.026726 17904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:41899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:41899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:52.028103 17904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:52.029404 17904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:52.040668 17909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:52.040724 17912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:52.040697 17910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:52.041893 17904 server_base.cc:1034] running on GCE node
I20260923 20:21:52.042677 17904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:52.043993 17904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:52.045229 17904 hybrid_clock.cc:648] HybridClock initialized: now 1790194912045186 us; error 30 us; skew 500 ppm
I20260923 20:21:52.045586 17904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:52 dist-test-slave-wmd0 krb5kdc[17891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194912, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:52.053962 17904 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:52.055594 17904 webserver.cc:460] Webserver started at http://127.14.127.254:41461/ using document root <none> and password file <none>
I20260923 20:21:52.056290 17904 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:52.056437 17904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:52.056757 17904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:52.059755 17904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907"
format_stamp: "Formatted at 2026-09-23 20:21:52 on dist-test-slave-wmd0"
server_key: "f5fc6835c42143de975d92277f68380d"
server_key_iv: "0a973483cf025d10c1f6ede34315ee80"
server_key_version: "encryptionkey@0"
I20260923 20:21:52.060510 17904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907"
format_stamp: "Formatted at 2026-09-23 20:21:52 on dist-test-slave-wmd0"
server_key: "f5fc6835c42143de975d92277f68380d"
server_key_iv: "0a973483cf025d10c1f6ede34315ee80"
server_key_version: "encryptionkey@0"
I20260923 20:21:52.065604 17904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 20:21:52.069052 17919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:52.070343 17904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:21:52.070529 17904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907"
format_stamp: "Formatted at 2026-09-23 20:21:52 on dist-test-slave-wmd0"
server_key: "f5fc6835c42143de975d92277f68380d"
server_key_iv: "0a973483cf025d10c1f6ede34315ee80"
server_key_version: "encryptionkey@0"
I20260923 20:21:52.070792 17904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:52.109656 17904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:52.113871 17904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:52.114197 17904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:52.133018 17904 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:41899
I20260923 20:21:52.133051 17970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:41899 every 8 connection(s)
I20260923 20:21:52.134389 17904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:52.138068 17971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:52.139509 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 17904
I20260923 20:21:52.139719 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:52.140345 14847 external_mini_cluster.cc:1428] Setting key dfd6421fee0b69f4bd77b80d55421227
I20260923 20:21:52.152289 17971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907: Bootstrap starting.
Sep 23 20:21:52 dist-test-slave-wmd0 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:52.155457 17971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:52.156672 17971 log.cc:824] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:52.160212 17971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907: No bootstrap required, opened a new log
I20260923 20:21:52.164374 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:52.142454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.14.127.254:41899)
0923 20:21:52.143309 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:52.143325 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:52.143376 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:21:52.145115 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:52.145146 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:52.145679 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:52.146092 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:52.147411 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:52.149475 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:52.150807 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:52.151815 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:52.152007 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:52.156064 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:52.156076 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:52.156094 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:52.156142 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:52.159981 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:52.160699 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:52.160707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:52.160715 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:52.160921 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:52.161297 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:52.161302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:52.161305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:52.161982 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:52.162314 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:52.162597 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 20:21:52.163061 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":388,"threads_started":1}
I20260923 20:21:52.169510 17971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 41899 } }
I20260923 20:21:52.169881 17971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:52.170025 17971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a4793fb4e514a7bbe7c70c8c1c9b907, State: Initialized, Role: FOLLOWER
I20260923 20:21:52.170567 17971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 41899 } }
I20260923 20:21:52.170815 17971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:52.170915 17971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:52.171039 17971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:52.173116 17971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 41899 } }
I20260923 20:21:52.173699 17971 leader_election.cc:302] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a4793fb4e514a7bbe7c70c8c1c9b907; no voters:
I20260923 20:21:52.174319 17971 leader_election.cc:288] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:52.174456 17976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:52.176375 17971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:52.181358 17976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [term 1 LEADER]: Becoming Leader. State: Replica: 6a4793fb4e514a7bbe7c70c8c1c9b907, State: Running, Role: LEADER
I20260923 20:21:52.181963 17976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 41899 } }
I20260923 20:21:52.184864 17977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 41899 } } }
I20260923 20:21:52.185213 17977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:52.185771 17978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a4793fb4e514a7bbe7c70c8c1c9b907. Latest consensus state: current_term: 1 leader_uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a4793fb4e514a7bbe7c70c8c1c9b907" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 41899 } } }
I20260923 20:21:52.186046 17978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:52.187791 17989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:52.191819 17989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:52.199282 17989 catalog_manager.cc:1347] Generated new cluster ID: 87c4f6888e1f4978bc321847224d7f17
I20260923 20:21:52.199393 17989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:52.220971 17989 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:52.222908 17989 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:52.232369 17989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a4793fb4e514a7bbe7c70c8c1c9b907: Generated new TSK 0
I20260923 20:21:52.233372 17989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:21:52.290321 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:41899
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:52.626102 17997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:52.626371 17997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:52.626513 17997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:52.633749 17997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:52.633843 17997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:52.634033 17997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:21:52.643474 17997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:52.644753 17997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:52.646042 17997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:52.658744 18003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:52.658893 18005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:52.659298 18002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:52.659802 17997 server_base.cc:1034] running on GCE node
I20260923 20:21:52.660387 17997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:52.661309 17997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:52.662535 17997 hybrid_clock.cc:648] HybridClock initialized: now 1790194912662485 us; error 62 us; skew 500 ppm
I20260923 20:21:52.662899 17997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:52 dist-test-slave-wmd0 krb5kdc[17891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194912, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:52.666043 17997 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:21:52.667495 17997 webserver.cc:460] Webserver started at http://127.14.127.193:42103/ using document root <none> and password file <none>
I20260923 20:21:52.668180 17997 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:52.668327 17997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:52.668638 17997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:52.672649 17997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "3da8b699837d46edb958d6e18da44f9f"
format_stamp: "Formatted at 2026-09-23 20:21:52 on dist-test-slave-wmd0"
server_key: "512db389d007716696ee5e69b7d84bff"
server_key_iv: "c453104285fcf911089d8a18dd925828"
server_key_version: "encryptionkey@0"
I20260923 20:21:52.673448 17997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "3da8b699837d46edb958d6e18da44f9f"
format_stamp: "Formatted at 2026-09-23 20:21:52 on dist-test-slave-wmd0"
server_key: "512db389d007716696ee5e69b7d84bff"
server_key_iv: "c453104285fcf911089d8a18dd925828"
server_key_version: "encryptionkey@0"
I20260923 20:21:52.678875 17997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 20:21:52.682276 18012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:52.683738 17997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:52.683925 17997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "3da8b699837d46edb958d6e18da44f9f"
format_stamp: "Formatted at 2026-09-23 20:21:52 on dist-test-slave-wmd0"
server_key: "512db389d007716696ee5e69b7d84bff"
server_key_iv: "c453104285fcf911089d8a18dd925828"
server_key_version: "encryptionkey@0"
I20260923 20:21:52.684139 17997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:52.703281 17997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:52.707801 17997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:52.708140 17997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:52.709079 17997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:52.710474 17997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:52.710583 17997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:52.710759 17997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:52.710843 17997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:52.743747 17997 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:39167
I20260923 20:21:52.743780 18124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:39167 every 8 connection(s)
I20260923 20:21:52.745039 17997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:21:52.750425 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 17997
I20260923 20:21:52.750681 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:21:52.751509 14847 external_mini_cluster.cc:1428] Setting key 7b0799a3fa2d5b4cbcc474439df261d5
Sep 23 20:21:52 dist-test-slave-wmd0 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194912, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:52.765846 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:52.747673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:48017 (local address 127.14.127.254:41899)
0923 20:21:52.748108 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:52.748113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:52.749065 (+ 952us) server_negotiation.cc:407] Connection header received
0923 20:21:52.751032 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:52.751039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:52.751127 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:52.751265 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:52.753071 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:52.754100 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:52.755770 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:52.756492 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:52.756635 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:52.760231 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:52.760236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:52.760239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:52.760271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:52.762889 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:52.763592 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:52.763599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:52.763607 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:52.763717 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:52.764155 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:52.764160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:52.764163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:52.764412 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:52.764572 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:52.765261 (+ 689us) server_negotiation.cc:299] Negotiation successful
0923 20:21:52.765475 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":227,"threads_started":1}
I20260923 20:21:52.766865 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:52.748204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:41899 (local address 127.14.127.193:48017)
0923 20:21:52.748906 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:52.748944 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:52.750721 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:52.751491 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:52.751498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:52.752095 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:52.752888 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:52.752917 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:52.754276 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:52.754280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:52.755611 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:52.755623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:52.756713 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:52.756718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:52.756918 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:52.757602 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:52.757628 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:52.760004 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:52.763027 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:52.763035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:52.763052 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:52.763461 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:52.763830 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:52.763835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:52.763839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:52.764035 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:52.764547 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:52.764555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:52.764892 (+ 337us) client_negotiation.cc:769] Sending connection context
0923 20:21:52.765159 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 20:21:52.765675 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":256,"threads_started":1}
I20260923 20:21:52.768567 18125 heartbeater.cc:347] Connected to a master server at 127.14.127.254:41899
I20260923 20:21:52.768857 18125 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:21:52.769580 18125 heartbeater.cc:511] Master 127.14.127.254:41899 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
I20260923 20:21:52.771826 17936 ts_manager.cc:194] Registered new tserver with Master: 3da8b699837d46edb958d6e18da44f9f (127.14.127.193:39167)
I20260923 20:21:52.773247 17936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:48017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:21:52.794261 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:41899
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:53.108103 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:53.108366 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:53.108525 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:53.115615 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:53.115715 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:53.115917 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:21:53.125447 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:53.126710 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:53.127964 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:53.141114 18137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:53.141173 18136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:53.141350 18139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:53.141876 18131 server_base.cc:1034] running on GCE node
I20260923 20:21:53.142467 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:53.143404 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:53.144647 18131 hybrid_clock.cc:648] HybridClock initialized: now 1790194913144590 us; error 48 us; skew 500 ppm
I20260923 20:21:53.144944 18131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:53.148164 18131 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:21:53.149641 18131 webserver.cc:460] Webserver started at http://127.14.127.194:43371/ using document root <none> and password file <none>
I20260923 20:21:53.150291 18131 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:53.150450 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:53.150818 18131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:53.153584 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "2df635fbdc26479d92e9e8c7e7c63935"
format_stamp: "Formatted at 2026-09-23 20:21:53 on dist-test-slave-wmd0"
server_key: "4afd404645a20c7e0908469d05d0ec1c"
server_key_iv: "c98dc3131125109b557ba088b7d92836"
server_key_version: "encryptionkey@0"
I20260923 20:21:53.154345 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "2df635fbdc26479d92e9e8c7e7c63935"
format_stamp: "Formatted at 2026-09-23 20:21:53 on dist-test-slave-wmd0"
server_key: "4afd404645a20c7e0908469d05d0ec1c"
server_key_iv: "c98dc3131125109b557ba088b7d92836"
server_key_version: "encryptionkey@0"
I20260923 20:21:53.159207 18131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:21:53.162492 18146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:53.163772 18131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:21:53.163931 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "2df635fbdc26479d92e9e8c7e7c63935"
format_stamp: "Formatted at 2026-09-23 20:21:53 on dist-test-slave-wmd0"
server_key: "4afd404645a20c7e0908469d05d0ec1c"
server_key_iv: "c98dc3131125109b557ba088b7d92836"
server_key_version: "encryptionkey@0"
I20260923 20:21:53.164126 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:53.194844 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:53.199146 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:53.199438 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:53.200384 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:53.201702 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:53.201774 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:53.201897 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:53.201956 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:53.246106 18131 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:41677
I20260923 20:21:53.246145 18258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:41677 every 8 connection(s)
I20260923 20:21:53.247524 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:21:53.252529 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 18131
I20260923 20:21:53.252730 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:21:53.253373 14847 external_mini_cluster.cc:1428] Setting key 60d76a6c6f88265423226cb72ffac636
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:53.270411 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:53.250936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:41899 (local address 127.14.127.194:54735)
0923 20:21:53.251737 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:53.251788 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:53.253789 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:53.254850 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:53.254861 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:53.255628 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:53.256238 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:53.256258 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.257313 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.257319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:53.258551 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:53.258564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.259397 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.259404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:53.259647 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:53.260333 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:53.260355 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:53.262999 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:53.266439 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:53.266449 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:53.266467 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:53.266968 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:53.267433 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:53.267438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:53.267442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:53.267725 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:53.268398 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:53.268406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:53.268816 (+ 410us) client_negotiation.cc:769] Sending connection context
0923 20:21:53.269029 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 20:21:53.269417 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":362,"threads_started":1}
I20260923 20:21:53.271526 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:53.250405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:54735 (local address 127.14.127.254:41899)
0923 20:21:53.250607 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:53.250635 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:53.251914 (+ 1279us) server_negotiation.cc:407] Connection header received
0923 20:21:53.254137 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:53.254146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:53.254243 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:53.254321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:53.256410 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.257154 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.258738 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.259168 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.259307 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:53.263255 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:53.263264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:53.263270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:53.263315 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:53.266289 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.267107 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.267114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.267117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.267309 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.267871 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.267878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.267883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.268230 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:53.270893 (+ 2663us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:53.271038 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 20:21:53.271288 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:21:53.272703 18259 heartbeater.cc:347] Connected to a master server at 127.14.127.254:41899
I20260923 20:21:53.273068 18259 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:53.273864 18259 heartbeater.cc:511] Master 127.14.127.254:41899 requested a full tablet report, sending...
I20260923 20:21:53.275559 17936 ts_manager.cc:194] Registered new tserver with Master: 2df635fbdc26479d92e9e8c7e7c63935 (127.14.127.194:41677)
I20260923 20:21:53.276522 17936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:54735
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:21:53.309371 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:41899
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:41619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:53.667800 18264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:53.668064 18264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:53.668156 18264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:53.676355 18264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:53.676497 18264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:53.676681 18264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:21:53.686576 18264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:41619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:53.687927 18264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:53.689195 18264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:53.702129 18269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:53.702096 18270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:53.702099 18272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:53.702733 18264 server_base.cc:1034] running on GCE node
I20260923 20:21:53.703492 18264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:53.704397 18264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:53.705622 18264 hybrid_clock.cc:648] HybridClock initialized: now 1790194913705581 us; error 53 us; skew 500 ppm
I20260923 20:21:53.705940 18264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:53.708994 18264 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:21:53.710526 18264 webserver.cc:460] Webserver started at http://127.14.127.195:35293/ using document root <none> and password file <none>
I20260923 20:21:53.711241 18264 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:53.711395 18264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:53.711688 18264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:53.714473 18264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "3eaee089d4004a669878ac86733e0fdb"
format_stamp: "Formatted at 2026-09-23 20:21:53 on dist-test-slave-wmd0"
server_key: "536127c774b6ad1741f2bf97e296376f"
server_key_iv: "4fc0cf5644ab3f40b75adb57061b2cf3"
server_key_version: "encryptionkey@0"
I20260923 20:21:53.715291 18264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "3eaee089d4004a669878ac86733e0fdb"
format_stamp: "Formatted at 2026-09-23 20:21:53 on dist-test-slave-wmd0"
server_key: "536127c774b6ad1741f2bf97e296376f"
server_key_iv: "4fc0cf5644ab3f40b75adb57061b2cf3"
server_key_version: "encryptionkey@0"
I20260923 20:21:53.720209 18264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:21:53.723610 18279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:53.724879 18264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:53.725049 18264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "3eaee089d4004a669878ac86733e0fdb"
format_stamp: "Formatted at 2026-09-23 20:21:53 on dist-test-slave-wmd0"
server_key: "536127c774b6ad1741f2bf97e296376f"
server_key_iv: "4fc0cf5644ab3f40b75adb57061b2cf3"
server_key_version: "encryptionkey@0"
I20260923 20:21:53.725235 18264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:53.751581 18264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:53.755872 18264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:53.756183 18264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:53.757105 18264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:53.758394 18264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:53.758496 18264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:53.758672 18264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:53.758754 18264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260923 20:21:53.776921 18125 heartbeater.cc:503] Master 127.14.127.254:41899 was elected leader, sending a full tablet report...
I20260923 20:21:53.792630 18264 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:44333
I20260923 20:21:53.792675 18391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:44333 every 8 connection(s)
I20260923 20:21:53.793885 18264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:21:53.800232 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 18264
I20260923 20:21:53.800423 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:21:53.800876 14847 external_mini_cluster.cc:1428] Setting key 794b0ded5e9c873d6bd895bdc8bc1d45
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:53.813172 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:53.796418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:48723 (local address 127.14.127.254:41899)
0923 20:21:53.796922 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:53.796926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:53.797680 (+ 754us) server_negotiation.cc:407] Connection header received
0923 20:21:53.799506 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:53.799511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:53.799602 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:53.799719 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:53.801325 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.802216 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.803893 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.804304 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.804438 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:53.807792 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:53.807797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:53.807800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:53.807832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:53.810145 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.810937 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.810941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.810944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.811078 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.811493 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.811502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.811505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.811761 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:53.811927 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:53.812591 (+ 664us) server_negotiation.cc:299] Negotiation successful
0923 20:21:53.812801 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":286,"threads_started":1}
I20260923 20:21:53.814054 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:53.796853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:41899 (local address 127.14.127.195:48723)
0923 20:21:53.797514 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:53.797565 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:53.799248 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:53.799961 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:53.799970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:53.800562 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:53.801161 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:53.801180 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.802364 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.802368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:53.803742 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:53.803760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.804525 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.804532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:53.804762 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:53.805356 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:53.805378 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:53.807495 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:53.810301 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:53.810309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:53.810324 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:53.810798 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:53.811187 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:53.811193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:53.811197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:53.811374 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:53.811889 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:53.811896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:53.812223 (+ 327us) client_negotiation.cc:769] Sending connection context
0923 20:21:53.812476 (+ 253us) client_negotiation.cc:240] Negotiation successful
0923 20:21:53.812974 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":254,"threads_started":1}
I20260923 20:21:53.815478 18392 heartbeater.cc:347] Connected to a master server at 127.14.127.254:41899
I20260923 20:21:53.815804 18392 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:53.816546 18392 heartbeater.cc:511] Master 127.14.127.254:41899 requested a full tablet report, sending...
I20260923 20:21:53.817817 17936 ts_manager.cc:194] Registered new tserver with Master: 3eaee089d4004a669878ac86733e0fdb (127.14.127.195:44333)
I20260923 20:21:53.818620 17936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:48723
I20260923 20:21:53.828233 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:53.873682 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:53.860167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58546 (local address 127.14.127.254:41899)
0923 20:21:53.860333 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:53.860337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:53.860816 (+ 479us) server_negotiation.cc:407] Connection header received
0923 20:21:53.860961 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:53.860966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:53.861030 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:53.861130 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:53.862480 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.863252 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.864437 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.864805 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.864893 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:53.868631 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:53.868636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:53.868639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:53.868670 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:53.871378 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.872010 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.872014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.872016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.872133 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.872438 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.872441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.872443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.872748 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:53.872936 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:53.873188 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 20:21:53.873338 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 20:21:53.874238 17936 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58546
I20260923 20:21:53.876529 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17997
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:21:53.891518 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:53.877516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.14.127.193:39167)
0923 20:21:53.878053 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:53.878060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:53.878084 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:21:53.878137 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:53.878144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:53.878344 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:53.878466 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:53.879605 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.880550 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.881656 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.882271 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.882388 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:53.886128 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:53.886138 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:53.886152 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:53.886190 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:53.889009 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.889644 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.889651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.889653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.889769 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.890087 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.890090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.890092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.890409 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:53.890720 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:53.890914 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 20:21:53.891111 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":246,"threads_started":1}
W20260923 20:21:53.892724 18120 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47368
W20260923 20:21:53.893800 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17997: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:53.893998 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 17997
I20260923 20:21:53.910130 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18131
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:21:53.925668 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:53.911053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.14.127.194:41677)
0923 20:21:53.911637 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:53.911644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:53.911669 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:21:53.911732 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:53.911739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:53.911944 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:53.912085 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:53.913228 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.914246 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.915574 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.916114 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.916210 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:53.920201 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:53.920213 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:53.920227 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:53.920259 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:53.923246 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.923817 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.923825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.923827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.923968 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.924279 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.924282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.924284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.924611 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:53.924816 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:53.925154 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 20:21:53.925347 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":323,"threads_started":1}
W20260923 20:21:53.926966 18254 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42028
W20260923 20:21:53.928071 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:53.928197 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 18131
I20260923 20:21:53.943526 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18264
Sep 23 20:21:53 dist-test-slave-wmd0 krb5kdc[17891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194913, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:21:53.958997 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:53.944513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.14.127.195:44333)
0923 20:21:53.945239 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:53.945246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:53.945275 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:21:53.945336 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:53.945342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:53.945542 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:53.945673 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:53.946951 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.948011 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.949142 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:53.949683 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:53.949782 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:53.953983 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:53.953994 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:53.954008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:53.954039 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:53.956683 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.957252 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.957258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.957260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.957369 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:53.957651 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:53.957654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:53.957656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:53.957963 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:53.958191 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:53.958395 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 20:21:53.958581 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":398,"threads_started":1}
W20260923 20:21:53.960246 18387 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46094
W20260923 20:21:53.961373 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18264: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:53.961560 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 18264
I20260923 20:21:53.976812 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 17904
I20260923 20:21:53.978020 17966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:54.086438 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 17904
2026-09-23T20:21:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3665 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:54 dist-test-slave-wmd0 krb5kdc[18410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:54 dist-test-slave-wmd0 krb5kdc[18410](info): set up 2 sockets
Sep 23 20:21:54 dist-test-slave-wmd0 krb5kdc[18410](info): commencing operation
krb5kdc: starting...
W20260923 20:21:55.167052 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:21:55 dist-test-slave-wmd0 krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:55.355132 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:35393
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:55.669262 18423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:55.669536 18423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:55.669691 18423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:55.676714 18423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:55.676811 18423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:55.676906 18423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:55.677009 18423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:55.677070 18423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:55.688295 18423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:55.689567 18423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:55.690896 18423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:55.702184 18428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:55.702399 18429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:55.702562 18431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:55.703430 18423 server_base.cc:1034] running on GCE node
I20260923 20:21:55.704100 18423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:55.705364 18423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:55.706564 18423 hybrid_clock.cc:648] HybridClock initialized: now 1790194915706517 us; error 37 us; skew 500 ppm
I20260923 20:21:55.706938 18423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:55 dist-test-slave-wmd0 krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194915, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:55.709885 18423 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:55.711313 18423 webserver.cc:460] Webserver started at http://127.14.127.254:37881/ using document root <none> and password file <none>
I20260923 20:21:55.711968 18423 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:55.712093 18423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:55.712409 18423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:55.715305 18423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "1be3c08c7bfb4a73b8242136e8befc33"
format_stamp: "Formatted at 2026-09-23 20:21:55 on dist-test-slave-wmd0"
server_key: "1eb5434539e46d873e0aab4540d83809"
server_key_iv: "bff47d3d7ddec01e02bfc28cd9cb557a"
server_key_version: "encryptionkey@0"
I20260923 20:21:55.716053 18423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "1be3c08c7bfb4a73b8242136e8befc33"
format_stamp: "Formatted at 2026-09-23 20:21:55 on dist-test-slave-wmd0"
server_key: "1eb5434539e46d873e0aab4540d83809"
server_key_iv: "bff47d3d7ddec01e02bfc28cd9cb557a"
server_key_version: "encryptionkey@0"
I20260923 20:21:55.720914 18423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:21:55.724308 18438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:55.725548 18423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:21:55.725720 18423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "1be3c08c7bfb4a73b8242136e8befc33"
format_stamp: "Formatted at 2026-09-23 20:21:55 on dist-test-slave-wmd0"
server_key: "1eb5434539e46d873e0aab4540d83809"
server_key_iv: "bff47d3d7ddec01e02bfc28cd9cb557a"
server_key_version: "encryptionkey@0"
I20260923 20:21:55.725924 18423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:55.747920 18423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:55.751987 18423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:55.752267 18423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:55.776340 18423 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:35393
I20260923 20:21:55.776358 18489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:35393 every 8 connection(s)
I20260923 20:21:55.777704 18423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:55.781368 18490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:55.782521 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 18423
I20260923 20:21:55.782779 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:55.783593 14847 external_mini_cluster.cc:1428] Setting key 349f696f13ce47ad1420816f6af21223
I20260923 20:21:55.794294 18490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33: Bootstrap starting.
Sep 23 20:21:55 dist-test-slave-wmd0 krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:55.797564 18490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:55.798825 18490 log.cc:824] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:55.802070 18490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33: No bootstrap required, opened a new log
I20260923 20:21:55.805783 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:55.785697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.14.127.254:35393)
0923 20:21:55.786464 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:55.786478 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:55.786520 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:21:55.788200 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:55.788224 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:55.788603 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:55.789000 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:55.790447 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:55.791838 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:55.793113 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:55.793777 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:55.793982 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:55.798057 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:55.798065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:55.798080 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:55.798118 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:55.801773 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:55.802595 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:55.802601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:55.802608 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:55.802825 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:55.803230 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:55.803233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:55.803235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:55.803729 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:55.803962 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:55.804406 (+ 444us) server_negotiation.cc:299] Negotiation successful
0923 20:21:55.804786 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":357,"threads_started":1}
I20260923 20:21:55.810214 18490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be3c08c7bfb4a73b8242136e8befc33" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35393 } }
I20260923 20:21:55.810566 18490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:55.810696 18490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be3c08c7bfb4a73b8242136e8befc33, State: Initialized, Role: FOLLOWER
I20260923 20:21:55.811300 18490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be3c08c7bfb4a73b8242136e8befc33" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35393 } }
I20260923 20:21:55.811535 18490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:55.811657 18490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:55.811832 18490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:55.813964 18490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be3c08c7bfb4a73b8242136e8befc33" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35393 } }
I20260923 20:21:55.814461 18490 leader_election.cc:302] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be3c08c7bfb4a73b8242136e8befc33; no voters:
I20260923 20:21:55.815147 18490 leader_election.cc:288] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:55.815297 18495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:55.815935 18495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [term 1 LEADER]: Becoming Leader. State: Replica: 1be3c08c7bfb4a73b8242136e8befc33, State: Running, Role: LEADER
I20260923 20:21:55.816496 18495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be3c08c7bfb4a73b8242136e8befc33" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35393 } }
I20260923 20:21:55.817236 18490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:55.819303 18497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be3c08c7bfb4a73b8242136e8befc33. Latest consensus state: current_term: 1 leader_uuid: "1be3c08c7bfb4a73b8242136e8befc33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be3c08c7bfb4a73b8242136e8befc33" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35393 } } }
I20260923 20:21:55.819756 18497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:55.820385 18496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be3c08c7bfb4a73b8242136e8befc33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be3c08c7bfb4a73b8242136e8befc33" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 35393 } } }
I20260923 20:21:55.820755 18496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:55.820861 18502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:55.828794 18502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:55.836727 18502 catalog_manager.cc:1347] Generated new cluster ID: ba11874a9eec43608d91865ebd6efb62
I20260923 20:21:55.836834 18502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:55.849866 18502 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:55.851707 18502 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:55.859402 18502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be3c08c7bfb4a73b8242136e8befc33: Generated new TSK 0
I20260923 20:21:55.860349 18502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:21:55.902936 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35393
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:56.223914 18516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:56.224262 18516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:56.224366 18516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:56.231709 18516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:56.231805 18516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:56.231922 18516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:21:56.241313 18516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:56.242512 18516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:56.243813 18516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:56.256675 18524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:56.256873 18521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:56.256839 18522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:56.257849 18516 server_base.cc:1034] running on GCE node
I20260923 20:21:56.258417 18516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:56.259344 18516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:56.260547 18516 hybrid_clock.cc:648] HybridClock initialized: now 1790194916260494 us; error 43 us; skew 500 ppm
I20260923 20:21:56.260975 18516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:56 dist-test-slave-wmd0 krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194916, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:56.264129 18516 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:21:56.265558 18516 webserver.cc:460] Webserver started at http://127.14.127.193:33125/ using document root <none> and password file <none>
I20260923 20:21:56.266222 18516 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:56.266361 18516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:56.266746 18516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:56.269688 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "9b5cafc44186496bbf7a26a964319933"
format_stamp: "Formatted at 2026-09-23 20:21:56 on dist-test-slave-wmd0"
server_key: "d9a260b41f5987ef36af2809b55eeedf"
server_key_iv: "cb3f94acd79ae53d3ec8764a524bbd1e"
server_key_version: "encryptionkey@0"
I20260923 20:21:56.270474 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "9b5cafc44186496bbf7a26a964319933"
format_stamp: "Formatted at 2026-09-23 20:21:56 on dist-test-slave-wmd0"
server_key: "d9a260b41f5987ef36af2809b55eeedf"
server_key_iv: "cb3f94acd79ae53d3ec8764a524bbd1e"
server_key_version: "encryptionkey@0"
I20260923 20:21:56.275398 18516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260923 20:21:56.278816 18531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:56.280184 18516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:21:56.280361 18516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "9b5cafc44186496bbf7a26a964319933"
format_stamp: "Formatted at 2026-09-23 20:21:56 on dist-test-slave-wmd0"
server_key: "d9a260b41f5987ef36af2809b55eeedf"
server_key_iv: "cb3f94acd79ae53d3ec8764a524bbd1e"
server_key_version: "encryptionkey@0"
I20260923 20:21:56.280588 18516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:56.303535 18516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:56.307696 18516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:56.308003 18516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:56.308931 18516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:56.310367 18516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:56.310436 18516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:56.310566 18516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:56.310700 18516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260923 20:21:56.343308 18516 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:36781
I20260923 20:21:56.343339 18643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:36781 every 8 connection(s)
I20260923 20:21:56.344729 18516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:21:56.350760 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 18516
I20260923 20:21:56.351003 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:21:56.351853 14847 external_mini_cluster.cc:1428] Setting key f3884a9e3573adc51c8502239f74c4f5
Sep 23 20:21:56 dist-test-slave-wmd0 krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194916, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:56.366545 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:56.347435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:43227 (local address 127.14.127.254:35393)
0923 20:21:56.347888 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:56.347894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:56.348881 (+ 987us) server_negotiation.cc:407] Connection header received
0923 20:21:56.350989 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:56.350993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:56.351091 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:56.351200 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:56.353115 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:56.353972 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:56.355567 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:56.355959 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:56.356116 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:56.359584 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:56.359593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:56.359600 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:56.359659 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:56.363272 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:56.363979 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:56.363988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:56.363994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:56.364157 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:56.364544 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:56.364551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:56.364557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:56.364953 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:56.365120 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:56.365778 (+ 658us) server_negotiation.cc:299] Negotiation successful
0923 20:21:56.366051 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":222,"threads_started":1}
I20260923 20:21:56.367149 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:56.348007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35393 (local address 127.14.127.193:43227)
0923 20:21:56.348724 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:56.348761 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:56.350697 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:56.351476 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:56.351488 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:56.352234 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:56.352939 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:56.352966 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:56.354141 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:56.354146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:56.355409 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:56.355421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:56.356222 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:56.356230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:56.356459 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:56.357159 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:56.357181 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:56.359362 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:56.363400 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:56.363406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:56.363418 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:56.363840 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:56.364259 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:56.364262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:56.364264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:56.364426 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:56.365083 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:56.365088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:56.365389 (+ 301us) client_negotiation.cc:769] Sending connection context
0923 20:21:56.365623 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 20:21:56.365995 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":272,"threads_started":1}
I20260923 20:21:56.368847 18644 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35393
I20260923 20:21:56.369166 18644 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:56.369838 18644 heartbeater.cc:511] Master 127.14.127.254:35393 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:21:56.372000 18455 ts_manager.cc:194] Registered new tserver with Master: 9b5cafc44186496bbf7a26a964319933 (127.14.127.193:36781)
I20260923 20:21:56.373396 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:43227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:21:56.397229 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35393
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:56.713336 18650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:56.713606 18650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:56.713697 18650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:56.720753 18650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:56.720856 18650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:56.720963 18650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:21:56.730731 18650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:56.731916 18650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:56.733152 18650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:56.746111 18658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:56.746070 18655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:56.746155 18656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:56.752218 18650 server_base.cc:1034] running on GCE node
I20260923 20:21:56.752882 18650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:56.753849 18650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:56.755081 18650 hybrid_clock.cc:648] HybridClock initialized: now 1790194916755028 us; error 47 us; skew 500 ppm
I20260923 20:21:56.755401 18650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:56 dist-test-slave-wmd0 krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194916, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:56.758467 18650 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:21:56.760032 18650 webserver.cc:460] Webserver started at http://127.14.127.194:46347/ using document root <none> and password file <none>
I20260923 20:21:56.760757 18650 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:56.760903 18650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:56.761217 18650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:56.764191 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "54c58681c59d448d9b8e872b51af4fe9"
format_stamp: "Formatted at 2026-09-23 20:21:56 on dist-test-slave-wmd0"
server_key: "58ea403940b7f2c91625715e97370073"
server_key_iv: "c3565ecd4e4101fd6c9388b6be16159d"
server_key_version: "encryptionkey@0"
I20260923 20:21:56.764990 18650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "54c58681c59d448d9b8e872b51af4fe9"
format_stamp: "Formatted at 2026-09-23 20:21:56 on dist-test-slave-wmd0"
server_key: "58ea403940b7f2c91625715e97370073"
server_key_iv: "c3565ecd4e4101fd6c9388b6be16159d"
server_key_version: "encryptionkey@0"
I20260923 20:21:56.770169 18650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:21:56.773622 18665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:56.775018 18650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:56.775202 18650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "54c58681c59d448d9b8e872b51af4fe9"
format_stamp: "Formatted at 2026-09-23 20:21:56 on dist-test-slave-wmd0"
server_key: "58ea403940b7f2c91625715e97370073"
server_key_iv: "c3565ecd4e4101fd6c9388b6be16159d"
server_key_version: "encryptionkey@0"
I20260923 20:21:56.775417 18650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:56.805250 18650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:56.809315 18650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:56.809628 18650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:56.810708 18650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:56.812057 18650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:56.812161 18650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:56.812314 18650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:56.812395 18650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:56.847445 18650 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:42335
I20260923 20:21:56.847534 18777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:42335 every 8 connection(s)
I20260923 20:21:56.848757 18650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:21:56.854745 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 18650
I20260923 20:21:56.854964 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:21:56.855410 14847 external_mini_cluster.cc:1428] Setting key 72c06a136a9dd8e33c0f5b74bd1d2a59
Sep 23 20:21:56 dist-test-slave-wmd0 krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194916, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:56.870024 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:56.851435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:51833 (local address 127.14.127.254:35393)
0923 20:21:56.851690 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:56.851696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:56.852791 (+ 1095us) server_negotiation.cc:407] Connection header received
0923 20:21:56.854743 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:56.854748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:56.854822 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:56.854963 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:56.856594 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:56.857563 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:56.859193 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:56.859988 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:56.860133 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:56.863712 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:56.863726 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:56.863729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:56.863762 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:56.866841 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:56.867569 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:56.867575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:56.867578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:56.867753 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:56.868305 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:56.868308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:56.868310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:56.868572 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:56.868677 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:56.869585 (+ 908us) server_negotiation.cc:299] Negotiation successful
0923 20:21:56.869749 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":105}
I20260923 20:21:56.871713 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:56.851946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35393 (local address 127.14.127.194:51833)
0923 20:21:56.852623 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:56.852660 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:56.854389 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:56.855218 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:56.855227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:56.855803 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:56.856418 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:56.856447 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:56.857705 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:56.857709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:56.859044 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:56.859056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:56.860199 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:56.860204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:56.860402 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:56.861096 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:56.861118 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:56.863487 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:56.866980 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:56.866989 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:56.867006 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:56.867433 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:56.867893 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:56.867901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:56.867908 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:56.868182 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:56.868701 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:56.868711 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:56.869193 (+ 482us) client_negotiation.cc:769] Sending connection context
0923 20:21:56.869504 (+ 311us) client_negotiation.cc:240] Negotiation successful
0923 20:21:56.870113 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":269,"threads_started":1}
I20260923 20:21:56.873310 18778 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35393
I20260923 20:21:56.873617 18778 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:56.874308 18778 heartbeater.cc:511] Master 127.14.127.254:35393 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
I20260923 20:21:56.875814 18455 ts_manager.cc:194] Registered new tserver with Master: 54c58681c59d448d9b8e872b51af4fe9 (127.14.127.194:42335)
I20260923 20:21:56.876705 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:51833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:21:56.899681 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35393
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:57.213301 18783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:57.213632 18783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:57.213814 18783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:57.220988 18783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:57.221130 18783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:57.221311 18783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:21:57.231266 18783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:35393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:57.232563 18783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:57.233865 18783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:57.246481 18788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:57.246654 18789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:57.246659 18791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:57.247795 18783 server_base.cc:1034] running on GCE node
I20260923 20:21:57.248409 18783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:57.249341 18783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:57.250566 18783 hybrid_clock.cc:648] HybridClock initialized: now 1790194917250520 us; error 58 us; skew 500 ppm
I20260923 20:21:57.250916 18783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194917, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:57.259871 18783 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:21:57.261601 18783 webserver.cc:460] Webserver started at http://127.14.127.195:42469/ using document root <none> and password file <none>
I20260923 20:21:57.262399 18783 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:57.262575 18783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:57.262948 18783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:57.266047 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "abe1b05e1be84113a39a176b3fea0afb"
format_stamp: "Formatted at 2026-09-23 20:21:57 on dist-test-slave-wmd0"
server_key: "59441bce28b95ae53b2475adc1e088f2"
server_key_iv: "c325f0b24a7b488d0c559842c856dd4b"
server_key_version: "encryptionkey@0"
I20260923 20:21:57.267007 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "abe1b05e1be84113a39a176b3fea0afb"
format_stamp: "Formatted at 2026-09-23 20:21:57 on dist-test-slave-wmd0"
server_key: "59441bce28b95ae53b2475adc1e088f2"
server_key_iv: "c325f0b24a7b488d0c559842c856dd4b"
server_key_version: "encryptionkey@0"
I20260923 20:21:57.272466 18783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:21:57.275990 18798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:57.277354 18783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:21:57.277549 18783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "abe1b05e1be84113a39a176b3fea0afb"
format_stamp: "Formatted at 2026-09-23 20:21:57 on dist-test-slave-wmd0"
server_key: "59441bce28b95ae53b2475adc1e088f2"
server_key_iv: "c325f0b24a7b488d0c559842c856dd4b"
server_key_version: "encryptionkey@0"
I20260923 20:21:57.277776 18783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:57.304152 18783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:57.308558 18783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:57.308861 18783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:57.309826 18783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:21:57.311223 18783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:21:57.311328 18783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:57.311476 18783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:21:57.311558 18783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:57.345180 18783 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:45055
I20260923 20:21:57.345211 18910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:45055 every 8 connection(s)
I20260923 20:21:57.346858 18783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:21:57.348222 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 18783
I20260923 20:21:57.348430 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:21:57.348979 14847 external_mini_cluster.cc:1428] Setting key 736e31e4029370cf110e5f87ebcaa2d8
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194917, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:57.367710 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:57.350229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:50517 (local address 127.14.127.254:35393)
0923 20:21:57.350430 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:57.350434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:57.352401 (+ 1967us) server_negotiation.cc:407] Connection header received
0923 20:21:57.354110 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:57.354114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:57.354175 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:57.354253 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:57.355728 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.356457 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.357959 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.358356 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.358445 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:57.361859 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:57.361864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:57.361867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:57.361898 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:57.365161 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.365800 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.365804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.365806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.365929 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.366278 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.366281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.366283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.366529 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:57.366709 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:57.367305 (+ 596us) server_negotiation.cc:299] Negotiation successful
0923 20:21:57.367465 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 20:21:57.368669 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:57.350880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:35393 (local address 127.14.127.195:50517)
0923 20:21:57.352246 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:57.352298 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:57.353862 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:57.354460 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:57.354467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:57.355038 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:57.355581 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:57.355597 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.356595 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.356599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:57.357830 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:57.357842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.358587 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.358592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:57.358851 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:57.359554 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:57.359575 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:57.361676 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:57.365273 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:57.365279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:57.365291 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:57.365694 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:57.366018 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:57.366021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:57.366023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:57.366182 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:57.366683 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:57.366689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:57.366996 (+ 307us) client_negotiation.cc:769] Sending connection context
0923 20:21:57.367211 (+ 215us) client_negotiation.cc:240] Negotiation successful
0923 20:21:57.367591 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"thread_start_us":332,"threads_started":1}
I20260923 20:21:57.370008 18911 heartbeater.cc:347] Connected to a master server at 127.14.127.254:35393
I20260923 20:21:57.370275 18911 heartbeater.cc:464] Registering TS with master...
I20260923 20:21:57.370958 18911 heartbeater.cc:511] Master 127.14.127.254:35393 requested a full tablet report, sending...
I20260923 20:21:57.372256 18455 ts_manager.cc:194] Registered new tserver with Master: abe1b05e1be84113a39a176b3fea0afb (127.14.127.195:45055)
I20260923 20:21:57.373016 18455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:50517
I20260923 20:21:57.376765 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:21:57.377028 18644 heartbeater.cc:503] Master 127.14.127.254:35393 was elected leader, sending a full tablet report...
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:21:57.424181 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:57.409110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.14.127.193:36781)
0923 20:21:57.409781 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:57.409788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:57.409812 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:21:57.409877 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:57.409884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:57.410101 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:57.410288 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:57.411807 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.412801 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.414046 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.414665 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.414815 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:57.418583 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:57.418594 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:57.418650 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:57.418700 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:57.421293 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.421984 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.421993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.421996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.422139 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.422496 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.422501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.422505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.422922 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:57.423248 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:57.423512 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 20:21:57.423786 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":311,"threads_started":1}
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:57.441689 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:57.428574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.14.127.254:35393)
0923 20:21:57.428798 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:57.428805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:57.428934 (+ 129us) server_negotiation.cc:407] Connection header received
0923 20:21:57.429045 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:57.429048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:57.429111 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:57.429189 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:57.430508 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.431277 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.432444 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.432839 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.432922 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:57.436599 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:57.436603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:57.436606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:57.436638 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:57.439270 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.439805 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.439809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.439811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.439928 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.440238 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.440241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.440243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.440719 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:57.440950 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:57.441183 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 20:21:57.441353 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 20:21:57.442279 18455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36218
I20260923 20:21:57.443289 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18516
I20260923 20:21:57.455346 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:57.444157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.14.127.193:36781)
0923 20:21:57.444322 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:57.444326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:57.444583 (+ 257us) server_negotiation.cc:407] Connection header received
0923 20:21:57.444700 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:57.444704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:57.444792 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:57.444870 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:57.446006 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.446752 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.447908 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.448350 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.448443 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:57.450477 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:57.450481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:57.450484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:57.450515 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:57.453302 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.453849 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.453852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.453855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.453963 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.454269 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.454272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.454274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.454493 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:57.454725 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:57.454961 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 20:21:57.455123 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:21:57.455952 18639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:57.538597 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 18516
I20260923 20:21:57.554581 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18650
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:21:57.569782 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:57.555491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.14.127.194:42335)
0923 20:21:57.556034 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:57.556042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:57.556068 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:21:57.556130 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:57.556136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:57.556357 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:57.556487 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:57.557587 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.558578 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.559849 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.560432 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.560542 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:57.564549 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:57.564560 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:57.564575 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:57.564607 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:57.567332 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.567925 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.567932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.567934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.568059 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.568468 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.568471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.568473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.568782 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:57.569002 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:57.569291 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 20:21:57.569482 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":290,"threads_started":1}
I20260923 20:21:57.571113 18773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:57.653872 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 18650
I20260923 20:21:57.669565 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18783
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:21:57.685592 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:57.670665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.14.127.195:45055)
0923 20:21:57.671286 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:57.671293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:57.671321 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:21:57.671379 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:57.671395 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:57.671601 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:57.671738 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:57.672998 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.674062 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.675459 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:57.676024 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:57.676131 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:57.680295 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:57.680306 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:57.680321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:57.680355 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:57.683255 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.683879 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.683885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.683888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.683999 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:57.684287 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:57.684290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:57.684292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:57.684628 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:57.684855 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:57.685103 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 20:21:57.685302 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":313,"threads_started":1}
I20260923 20:21:57.686964 18906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:57.773109 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 18783
I20260923 20:21:57.788791 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18423
I20260923 20:21:57.789979 18485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:57.894172 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 18423
2026-09-23T20:21:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3808 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18930](info): set up 2 sockets
Sep 23 20:21:57 dist-test-slave-wmd0 krb5kdc[18930](info): commencing operation
krb5kdc: starting...
W20260923 20:21:58.979178 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:21:59 dist-test-slave-wmd0 krb5kdc[18930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:21:59.166170 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:42365
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:42365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:59.492132 18943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:59.492419 18943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:59.492578 18943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:59.499791 18943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:59.499902 18943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:59.499948 18943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:59.500057 18943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:59.500142 18943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:59.510066 18943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:42365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:42365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:59.511433 18943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:59.512733 18943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:59.524262 18948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:59.524530 18951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:59.524471 18949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:59.525506 18943 server_base.cc:1034] running on GCE node
I20260923 20:21:59.526347 18943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:59.527848 18943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:59.529119 18943 hybrid_clock.cc:648] HybridClock initialized: now 1790194919529064 us; error 58 us; skew 500 ppm
I20260923 20:21:59.529544 18943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:59 dist-test-slave-wmd0 krb5kdc[18930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194919, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:59.532636 18943 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:21:59.533890 18943 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:59.534026 18943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:59.534364 18943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:59.537387 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a"
format_stamp: "Formatted at 2026-09-23 20:21:59 on dist-test-slave-wmd0"
server_key: "37c6c5f69f43207b1b78a22f142d6015"
server_key_iv: "67b2ff9d62123c44f06f1ccb5d73e4e4"
server_key_version: "encryptionkey@0"
I20260923 20:21:59.538166 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a"
format_stamp: "Formatted at 2026-09-23 20:21:59 on dist-test-slave-wmd0"
server_key: "37c6c5f69f43207b1b78a22f142d6015"
server_key_iv: "67b2ff9d62123c44f06f1ccb5d73e4e4"
server_key_version: "encryptionkey@0"
I20260923 20:21:59.543128 18943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 20:21:59.546411 18957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:59.547746 18943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:59.547945 18943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a"
format_stamp: "Formatted at 2026-09-23 20:21:59 on dist-test-slave-wmd0"
server_key: "37c6c5f69f43207b1b78a22f142d6015"
server_key_iv: "67b2ff9d62123c44f06f1ccb5d73e4e4"
server_key_version: "encryptionkey@0"
I20260923 20:21:59.548162 18943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:59.577617 18943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:59.581686 18943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:59.581987 18943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:59.600760 18943 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:42365
I20260923 20:21:59.600795 19008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:42365 every 8 connection(s)
I20260923 20:21:59.601676 18943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:59.603253 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 18943
I20260923 20:21:59.603492 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:21:59.604108 14847 external_mini_cluster.cc:1428] Setting key 1decefdcb5690a51315288053e074a3f
I20260923 20:21:59.605825 19009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:21:59 dist-test-slave-wmd0 krb5kdc[18930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:21:59.619395 19009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a: Bootstrap starting.
I20260923 20:21:59.623598 19009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:59.624898 19009 log.cc:824] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:59.628244 19009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a: No bootstrap required, opened a new log
I20260923 20:21:59.628576 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:59.606181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.14.127.254:42365)
0923 20:21:59.607055 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:59.607070 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:59.607117 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:21:59.609096 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:59.609123 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:59.609645 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:59.610094 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:59.611525 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:59.613527 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:59.614810 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:59.615857 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:59.616076 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:59.620203 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:59.620215 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:59.620233 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:59.620275 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:59.624147 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:59.624834 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:59.624846 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:59.624856 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:59.625078 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:59.625462 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:59.625469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:59.625475 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:59.626085 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:59.626391 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:59.626776 (+ 385us) server_negotiation.cc:299] Negotiation successful
0923 20:21:59.627237 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":404,"threads_started":1}
I20260923 20:21:59.636211 19009 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42365 } }
I20260923 20:21:59.636539 19009 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:59.636613 19009 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f3c5c1a0504012aa4996fdaf82dc4a, State: Initialized, Role: FOLLOWER
I20260923 20:21:59.637184 19009 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42365 } }
I20260923 20:21:59.637385 19009 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:59.637487 19009 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:59.637609 19009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:59.639696 19009 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42365 } }
I20260923 20:21:59.640123 19009 leader_election.cc:302] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0f3c5c1a0504012aa4996fdaf82dc4a; no voters:
I20260923 20:21:59.640609 19009 leader_election.cc:288] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:59.640758 19014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:59.641443 19014 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [term 1 LEADER]: Becoming Leader. State: Replica: c0f3c5c1a0504012aa4996fdaf82dc4a, State: Running, Role: LEADER
I20260923 20:21:59.641950 19014 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42365 } }
I20260923 20:21:59.642880 19009 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:59.644677 19015 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42365 } } }
I20260923 20:21:59.644989 19015 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:59.645835 19020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:59.645704 19016 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0f3c5c1a0504012aa4996fdaf82dc4a. Latest consensus state: current_term: 1 leader_uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f3c5c1a0504012aa4996fdaf82dc4a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42365 } } }
I20260923 20:21:59.647986 19016 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:59.651938 19020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:59.660419 19020 catalog_manager.cc:1347] Generated new cluster ID: c6f8c83b4d4146ee8507193115036bf9
I20260923 20:21:59.660537 19020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:59.686105 19020 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:59.687564 19020 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:59.698835 19020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a: Generated new TSK 0
I20260923 20:21:59.699855 19020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:21:59.759141 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42365
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:00.081781 19035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:00.082094 19035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:00.082263 19035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:00.089458 19035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:00.089599 19035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:00.089776 19035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:00.099188 19035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:00.100417 19035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:00.101675 19035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:00.114037 19041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:00.114080 19043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:00.114109 19040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:00.115056 19035 server_base.cc:1034] running on GCE node
I20260923 20:22:00.115625 19035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:00.116504 19035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:00.117733 19035 hybrid_clock.cc:648] HybridClock initialized: now 1790194920117693 us; error 53 us; skew 500 ppm
I20260923 20:22:00.118049 19035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:00 dist-test-slave-wmd0 krb5kdc[18930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:00.121025 19035 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:22:00.122174 19035 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:00.122306 19035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:00.122653 19035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:00.125411 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "eed9c09b0d76433ab67271e6d29f4f68"
format_stamp: "Formatted at 2026-09-23 20:22:00 on dist-test-slave-wmd0"
server_key: "5955dff155692d8643e1ebb5a292a9fd"
server_key_iv: "b92b3c0f83e02b658ac233924f572a15"
server_key_version: "encryptionkey@0"
I20260923 20:22:00.126202 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "eed9c09b0d76433ab67271e6d29f4f68"
format_stamp: "Formatted at 2026-09-23 20:22:00 on dist-test-slave-wmd0"
server_key: "5955dff155692d8643e1ebb5a292a9fd"
server_key_iv: "b92b3c0f83e02b658ac233924f572a15"
server_key_version: "encryptionkey@0"
I20260923 20:22:00.131095 19035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 20:22:00.134420 19049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:00.135696 19035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:00.135867 19035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "eed9c09b0d76433ab67271e6d29f4f68"
format_stamp: "Formatted at 2026-09-23 20:22:00 on dist-test-slave-wmd0"
server_key: "5955dff155692d8643e1ebb5a292a9fd"
server_key_iv: "b92b3c0f83e02b658ac233924f572a15"
server_key_version: "encryptionkey@0"
I20260923 20:22:00.136090 19035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:00.157677 19035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:00.161300 19035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:00.161561 19035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:00.162454 19035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:00.163761 19035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:00.163882 19035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:00.164016 19035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:00.164103 19035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:00.197257 19035 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:44567
I20260923 20:22:00.197340 19161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:44567 every 8 connection(s)
I20260923 20:22:00.198175 19035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:00.198599 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 19035
I20260923 20:22:00.198891 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:00.199704 14847 external_mini_cluster.cc:1428] Setting key 737ff5db7f4307ac69cbc19f88b883d7
Sep 23 20:22:00 dist-test-slave-wmd0 krb5kdc[18930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:22:00.220777 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:00.200812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:34151 (local address 127.14.127.254:42365)
0923 20:22:00.201281 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:00.201291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:00.202504 (+ 1213us) server_negotiation.cc:407] Connection header received
0923 20:22:00.205410 (+ 2906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:00.205414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:00.205485 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:00.205568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:00.207742 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:00.208701 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:00.210224 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:00.211004 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:00.211143 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:00.214638 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:00.214647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:00.214655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:00.214699 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:00.217465 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:00.218379 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:00.218383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:00.218385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:00.218498 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:00.219001 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:00.219005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:00.219007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:00.219275 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:00.219385 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:00.220245 (+ 860us) server_negotiation.cc:299] Negotiation successful
0923 20:22:00.220408 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":245,"threads_started":1}
I20260923 20:22:00.221714 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:00.201450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42365 (local address 127.14.127.193:34151)
0923 20:22:00.202312 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:00.202370 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:00.205099 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:00.205833 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:00.205847 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:00.206691 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:00.207560 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:00.207591 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:00.208861 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:00.208864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:00.210068 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:00.210080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:00.211199 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:00.211204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:00.211385 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:00.212076 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:00.212099 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:00.214396 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:00.217607 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:00.217616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:00.217633 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:00.218264 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:00.218641 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:00.218648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:00.218655 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:00.218890 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:00.219404 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:00.219412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:00.219871 (+ 459us) client_negotiation.cc:769] Sending connection context
0923 20:22:00.220128 (+ 257us) client_negotiation.cc:240] Negotiation successful
0923 20:22:00.220585 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":376,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
I20260923 20:22:00.223475 19162 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42365
I20260923 20:22:00.223770 19162 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:00.224373 19162 heartbeater.cc:511] Master 127.14.127.254:42365 requested a full tablet report, sending...
I20260923 20:22:00.226401 18974 ts_manager.cc:194] Registered new tserver with Master: eed9c09b0d76433ab67271e6d29f4f68 (127.14.127.193:44567)
I20260923 20:22:00.227845 18974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:34151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:22:00.246457 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42365
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:00.556648 19168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:00.556900 19168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:00.557051 19168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:00.564280 19168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:00.564376 19168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:00.564574 19168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:00.574424 19168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:00.575685 19168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:00.576928 19168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:00.589417 19173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:00.589479 19174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:00.589500 19176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:00.590446 19168 server_base.cc:1034] running on GCE node
I20260923 20:22:00.591109 19168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:00.591938 19168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:00.593127 19168 hybrid_clock.cc:648] HybridClock initialized: now 1790194920593082 us; error 43 us; skew 500 ppm
I20260923 20:22:00.593427 19168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:00 dist-test-slave-wmd0 krb5kdc[18930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:00.596447 19168 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:22:00.597551 19168 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:00.597635 19168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:00.597872 19168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:00.600745 19168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "b6b8b7c5cee04854997b134b49cf3410"
format_stamp: "Formatted at 2026-09-23 20:22:00 on dist-test-slave-wmd0"
server_key: "774d03cec0abcccbd63f1c9cdc78d384"
server_key_iv: "794b8a3a9e0967b3edb937a8a93a53ce"
server_key_version: "encryptionkey@0"
I20260923 20:22:00.601532 19168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "b6b8b7c5cee04854997b134b49cf3410"
format_stamp: "Formatted at 2026-09-23 20:22:00 on dist-test-slave-wmd0"
server_key: "774d03cec0abcccbd63f1c9cdc78d384"
server_key_iv: "794b8a3a9e0967b3edb937a8a93a53ce"
server_key_version: "encryptionkey@0"
I20260923 20:22:00.606436 19168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:22:00.609860 19182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:00.611204 19168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:00.611356 19168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "b6b8b7c5cee04854997b134b49cf3410"
format_stamp: "Formatted at 2026-09-23 20:22:00 on dist-test-slave-wmd0"
server_key: "774d03cec0abcccbd63f1c9cdc78d384"
server_key_iv: "794b8a3a9e0967b3edb937a8a93a53ce"
server_key_version: "encryptionkey@0"
I20260923 20:22:00.611579 19168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:00.653864 19168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:00.657882 19168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:00.658171 19168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:00.659179 19168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:00.660548 19168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:00.660636 19168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:00.660789 19168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:00.660881 19168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:00.694547 19168 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:36689
I20260923 20:22:00.694597 19294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:36689 every 8 connection(s)
I20260923 20:22:00.695436 19168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:00.704614 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 19168
I20260923 20:22:00.704821 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:00.705354 14847 external_mini_cluster.cc:1428] Setting key 5d6729e4ea81e6e1fc1536b6f652f9ae
Sep 23 20:22:00 dist-test-slave-wmd0 krb5kdc[18930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:00.716684 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:00.697840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:45553 (local address 127.14.127.254:42365)
0923 20:22:00.698013 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:00.698018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:00.699274 (+ 1256us) server_negotiation.cc:407] Connection header received
0923 20:22:00.701578 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:00.701583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:00.701677 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:00.701793 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:00.703735 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:00.704439 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:00.706509 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:00.707029 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:00.707174 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:00.711043 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:00.711049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:00.711052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:00.711083 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:00.713589 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:00.714354 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:00.714366 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:00.714369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:00.714493 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:00.714986 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:00.714993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:00.715001 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:00.715247 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:00.715405 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:00.716099 (+ 694us) server_negotiation.cc:299] Negotiation successful
0923 20:22:00.716325 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:22:00.717618 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:00.698226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42365 (local address 127.14.127.194:45553)
0923 20:22:00.699045 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:00.699101 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:00.701250 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:00.702074 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:00.702085 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:00.702948 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:00.703566 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:00.703583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:00.704612 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:00.704617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:00.706366 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:00.706383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:00.707319 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:00.707328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:00.707596 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:00.708273 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:00.708298 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:00.710823 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:00.713750 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:00.713760 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:00.713776 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:00.714225 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:00.714654 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:00.714659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:00.714666 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:00.714865 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:00.715378 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:00.715386 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:00.715738 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 20:22:00.715993 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 20:22:00.716515 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":320,"threads_started":1}
I20260923 20:22:00.719151 19295 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42365
I20260923 20:22:00.719455 19295 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:00.720088 19295 heartbeater.cc:511] Master 127.14.127.254:42365 requested a full tablet report, sending...
I20260923 20:22:00.721654 18974 ts_manager.cc:194] Registered new tserver with Master: b6b8b7c5cee04854997b134b49cf3410 (127.14.127.194:36689)
I20260923 20:22:00.722913 18974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:45553
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:22:00.749459 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42365
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:01.070231 19300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:01.070506 19300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:01.070601 19300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:01.077833 19300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:01.077936 19300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:01.078050 19300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:01.087805 19300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:01.089031 19300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:01.090255 19300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:01.102751 19305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:01.102774 19306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:01.102993 19308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:01.103561 19300 server_base.cc:1034] running on GCE node
I20260923 20:22:01.104092 19300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:01.104924 19300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:01.106122 19300 hybrid_clock.cc:648] HybridClock initialized: now 1790194921106062 us; error 49 us; skew 500 ppm
I20260923 20:22:01.106443 19300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:01 dist-test-slave-wmd0 krb5kdc[18930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194921, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:01.109599 19300 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:22:01.110795 19300 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:01.110927 19300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:01.111243 19300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:01.114135 19300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "72c142d050394960928d061a66207630"
format_stamp: "Formatted at 2026-09-23 20:22:01 on dist-test-slave-wmd0"
server_key: "ac5a8eba1c5f0e56ff97aedc4bfd74e3"
server_key_iv: "7ac23959608bd4c86da999d8cf31c93b"
server_key_version: "encryptionkey@0"
I20260923 20:22:01.115041 19300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "72c142d050394960928d061a66207630"
format_stamp: "Formatted at 2026-09-23 20:22:01 on dist-test-slave-wmd0"
server_key: "ac5a8eba1c5f0e56ff97aedc4bfd74e3"
server_key_iv: "7ac23959608bd4c86da999d8cf31c93b"
server_key_version: "encryptionkey@0"
I20260923 20:22:01.120169 19300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:22:01.123891 19314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:01.125201 19300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:01.125368 19300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "72c142d050394960928d061a66207630"
format_stamp: "Formatted at 2026-09-23 20:22:01 on dist-test-slave-wmd0"
server_key: "ac5a8eba1c5f0e56ff97aedc4bfd74e3"
server_key_iv: "7ac23959608bd4c86da999d8cf31c93b"
server_key_version: "encryptionkey@0"
I20260923 20:22:01.125594 19300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:01.152820 19300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:01.157320 19300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:01.157600 19300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:01.158663 19300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:01.160005 19300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:01.160127 19300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:01.160250 19300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:01.160334 19300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:01.193250 19300 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:46457
I20260923 20:22:01.193336 19426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:46457 every 8 connection(s)
I20260923 20:22:01.194118 19300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:01.198133 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 19300
I20260923 20:22:01.198323 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:01.198853 14847 external_mini_cluster.cc:1428] Setting key 8670a4903675247cd5bd84f661d75ec9
Sep 23 20:22:01 dist-test-slave-wmd0 krb5kdc[18930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194921, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:01.214121 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.196824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:58345 (local address 127.14.127.254:42365)
0923 20:22:01.197030 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.197037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.198256 (+ 1219us) server_negotiation.cc:407] Connection header received
0923 20:22:01.200557 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.200566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.200655 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.200771 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:01.202533 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.203452 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.204922 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.205323 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.205424 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.208777 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:01.208782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:01.208785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:01.208817 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:01.211196 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.211843 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.211846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.211849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.211976 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.212426 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.212430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.212433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.212768 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:01.212944 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.213575 (+ 631us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.213788 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:22:01.215333 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.197350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42365 (local address 127.14.127.195:58345)
0923 20:22:01.198106 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.198150 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:01.200200 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:01.201039 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:01.201050 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:01.201567 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:01.202351 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.202371 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.203589 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.203593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.204770 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.204781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.205503 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.205507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.205714 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.206308 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:01.206329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:01.208591 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:01.211322 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:01.211328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:01.211345 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:01.211744 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:01.212103 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:01.212107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:01.212109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:01.212307 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:01.212918 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:01.212924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:01.213220 (+ 296us) client_negotiation.cc:769] Sending connection context
0923 20:22:01.213474 (+ 254us) client_negotiation.cc:240] Negotiation successful
0923 20:22:01.214003 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":324,"threads_started":1}
I20260923 20:22:01.216634 19427 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42365
I20260923 20:22:01.216924 19427 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:01.217535 19427 heartbeater.cc:511] Master 127.14.127.254:42365 requested a full tablet report, sending...
I20260923 20:22:01.218930 18974 ts_manager.cc:194] Registered new tserver with Master: 72c142d050394960928d061a66207630 (127.14.127.195:46457)
I20260923 20:22:01.219727 18974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:58345
I20260923 20:22:01.227172 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:22:01.231382 19162 heartbeater.cc:503] Master 127.14.127.254:42365 was elected leader, sending a full tablet report...
I20260923 20:22:01.244688 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.232006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.14.127.254:42365)
0923 20:22:01.232189 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.232195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.232711 (+ 516us) server_negotiation.cc:407] Connection header received
0923 20:22:01.232940 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.232945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.233018 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.233122 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:01.234649 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.235362 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.236690 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.237111 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.237254 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.239569 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:01.239575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:01.239578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:01.239633 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:01.241975 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.242587 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.242594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.242597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.242790 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.243185 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.243190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.243194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.243563 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:01.243812 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.244147 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.244357 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:22:01.248401 18974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:01.255638 18974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:01.291442 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.268475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.14.127.194:36689)
0923 20:22:01.281467 (+ 12992us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.281481 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.282320 (+ 839us) server_negotiation.cc:407] Connection header received
0923 20:22:01.282420 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.282426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.282760 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.282991 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:01.283980 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.285488 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.288481 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.289900 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.290076 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.290232 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.290964 (+ 732us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.291128 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12741,"thread_start_us":13024,"threads_started":1}
I20260923 20:22:01.291518 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.268635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46457 (local address 127.0.0.1:57094)
0923 20:22:01.283087 (+ 14452us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.283117 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:01.283374 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:01.284045 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:01.284052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:01.284076 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:01.284475 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.284486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.286478 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.286483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.288274 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.288287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.290201 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.290209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.290534 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.290670 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 20:22:01.290808 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 20:22:01.291023 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14270,"mutex_wait_us":206,"spinlock_wait_cycles":18432,"thread_start_us":14123,"threads_started":1}
I20260923 20:22:01.291783 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.267890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.14.127.193:44567)
0923 20:22:01.284022 (+ 16132us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.284037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.284063 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:22:01.284176 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.284185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.284418 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.284632 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:01.285231 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.286738 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.289263 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.290557 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.290864 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.291025 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.291261 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.291414 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15941,"thread_start_us":15774,"threads_started":1}
I20260923 20:22:01.291805 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.267562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44567 (local address 127.0.0.1:55278)
0923 20:22:01.268658 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.268706 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:01.268881 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:01.284626 (+ 15745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:01.284631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:01.284657 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:01.285041 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.285053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.286885 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.286890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.289080 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.289094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.290869 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.290875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.291055 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.291087 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:22:01.291210 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 20:22:01.291345 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":1,"thread_start_us":14566,"threads_started":1}
I20260923 20:22:01.292513 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.269363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.14.127.195:46457)
0923 20:22:01.282246 (+ 12883us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.282252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.283242 (+ 990us) server_negotiation.cc:407] Connection header received
0923 20:22:01.283584 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.283593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.283839 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.284028 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:01.284704 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.286323 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.288437 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.289641 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.289824 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.290016 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.291975 (+ 1959us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.292169 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12727,"thread_start_us":12534,"threads_started":1}
I20260923 20:22:01.293205 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.267851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:36689 (local address 127.0.0.1:57140)
0923 20:22:01.281793 (+ 13942us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.281824 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:01.282013 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:01.283289 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:01.283299 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:01.283358 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:01.283750 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.283762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.285668 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.285673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.288305 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.288319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.290219 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.290224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.290539 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.290670 (+ 131us) client_negotiation.cc:769] Sending connection context
0923 20:22:01.290808 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 20:22:01.292817 (+ 2009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13767,"mutex_wait_us":1942,"spinlock_wait_cycles":2304,"thread_start_us":804,"threads_started":1}
I20260923 20:22:01.295945 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b (DEFAULT_TABLE table=test-table [id=e6d744af48384b5389ac352089d2c2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:01.295933 19362 tablet_service.cc:1467] Processing CreateTablet for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b (DEFAULT_TABLE table=test-table [id=e6d744af48384b5389ac352089d2c2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:01.297425 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:01.297457 19362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:01.298060 19230 tablet_service.cc:1467] Processing CreateTablet for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b (DEFAULT_TABLE table=test-table [id=e6d744af48384b5389ac352089d2c2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:01.299371 19230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:01.309536 19445 tablet_bootstrap.cc:492] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: Bootstrap starting.
I20260923 20:22:01.312505 19446 tablet_bootstrap.cc:492] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Bootstrap starting.
I20260923 20:22:01.313082 19445 tablet_bootstrap.cc:654] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:01.314494 19445 log.cc:824] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:01.316147 19446 tablet_bootstrap.cc:654] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:01.316689 19447 tablet_bootstrap.cc:492] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Bootstrap starting.
I20260923 20:22:01.317317 19446 log.cc:824] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:01.317879 19445 tablet_bootstrap.cc:492] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: No bootstrap required, opened a new log
I20260923 20:22:01.318133 19445 ts_tablet_manager.cc:1397] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:22:01.319589 19447 tablet_bootstrap.cc:654] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:01.320366 19446 tablet_bootstrap.cc:492] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: No bootstrap required, opened a new log
I20260923 20:22:01.320585 19447 log.cc:824] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:01.320634 19446 ts_tablet_manager.cc:1397] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 20:22:01.324815 19447 tablet_bootstrap.cc:492] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: No bootstrap required, opened a new log
I20260923 20:22:01.325078 19447 ts_tablet_manager.cc:1397] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260923 20:22:01.328574 19445 raft_consensus.cc:348] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.328995 19445 raft_consensus.cc:374] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:01.329111 19445 raft_consensus.cc:729] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed9c09b0d76433ab67271e6d29f4f68, State: Initialized, Role: FOLLOWER
I20260923 20:22:01.329725 19445 consensus_queue.cc:260] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.331153 19446 raft_consensus.cc:348] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.331666 19446 raft_consensus.cc:374] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:01.331761 19446 raft_consensus.cc:729] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c142d050394960928d061a66207630, State: Initialized, Role: FOLLOWER
I20260923 20:22:01.332427 19446 consensus_queue.cc:260] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.332856 19445 ts_tablet_manager.cc:1428] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260923 20:22:01.333770 19427 heartbeater.cc:503] Master 127.14.127.254:42365 was elected leader, sending a full tablet report...
I20260923 20:22:01.333581 19447 raft_consensus.cc:348] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.333971 19447 raft_consensus.cc:374] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:01.334069 19447 raft_consensus.cc:729] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b8b7c5cee04854997b134b49cf3410, State: Initialized, Role: FOLLOWER
I20260923 20:22:01.334371 19446 ts_tablet_manager.cc:1428] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 20:22:01.334859 19447 consensus_queue.cc:260] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.335940 19295 heartbeater.cc:503] Master 127.14.127.254:42365 was elected leader, sending a full tablet report...
I20260923 20:22:01.337430 19447 ts_tablet_manager.cc:1428] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
W20260923 20:22:01.445786 19428 tablet.cc:2369] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:22:01.448056 19296 tablet.cc:2369] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:22:01.448414 19451 raft_consensus.cc:482] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:01.448712 19451 raft_consensus.cc:504] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.450742 19451 leader_election.cc:288] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6b8b7c5cee04854997b134b49cf3410 (127.14.127.194:36689), 72c142d050394960928d061a66207630 (127.14.127.195:46457)
W20260923 20:22:01.451313 19163 tablet.cc:2369] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:22:01.457939 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.451306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:43363 (local address 127.14.127.194:36689)
0923 20:22:01.452316 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.452323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.452457 (+ 134us) server_negotiation.cc:407] Connection header received
0923 20:22:01.452734 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.452739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.452818 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.452923 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:01.453581 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.454273 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.456207 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.456953 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.457078 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.457177 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.457544 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.457672 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869}
I20260923 20:22:01.457973 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.450998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:33721 (local address 127.14.127.195:46457)
0923 20:22:01.451232 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.451237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.452190 (+ 953us) server_negotiation.cc:407] Connection header received
0923 20:22:01.452518 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.452523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.452601 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.452693 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:01.453472 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.454233 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.456241 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.457027 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.457193 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.457266 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.457575 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.457694 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:22:01.458213 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.450887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46457 (local address 127.14.127.193:33721)
0923 20:22:01.451997 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.452086 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:01.452317 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:01.452726 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:01.452731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:01.452758 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:01.453284 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.453298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.454365 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.454373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.456045 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.456056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.457260 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.457265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.457404 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.457426 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:22:01.457542 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:22:01.457665 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":98,"thread_start_us":1162,"threads_started":1}
I20260923 20:22:01.458261 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.451014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:36689 (local address 127.14.127.193:43363)
0923 20:22:01.452335 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.452349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:01.452561 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:01.452984 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:01.452987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:01.453008 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:01.453422 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.453435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.454435 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.454440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.456033 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:01.456045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.457175 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.457181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:01.457357 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.457408 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:22:01.457484 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 20:22:01.457629 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"mutex_wait_us":48,"thread_start_us":563,"threads_started":1}
I20260923 20:22:01.459266 19382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b5398a1ca4f65a8ebcbcb39f3dc2b" candidate_uuid: "eed9c09b0d76433ab67271e6d29f4f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c142d050394960928d061a66207630" is_pre_election: true
I20260923 20:22:01.459792 19382 raft_consensus.cc:2454] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed9c09b0d76433ab67271e6d29f4f68 in term 0.
I20260923 20:22:01.460426 19051 leader_election.cc:302] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c142d050394960928d061a66207630, eed9c09b0d76433ab67271e6d29f4f68; no voters:
I20260923 20:22:01.460505 19250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b5398a1ca4f65a8ebcbcb39f3dc2b" candidate_uuid: "eed9c09b0d76433ab67271e6d29f4f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b8b7c5cee04854997b134b49cf3410" is_pre_election: true
I20260923 20:22:01.460813 19451 raft_consensus.cc:2781] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:01.460934 19451 raft_consensus.cc:482] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:01.461072 19451 raft_consensus.cc:3037] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:01.461067 19250 raft_consensus.cc:2454] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed9c09b0d76433ab67271e6d29f4f68 in term 0.
I20260923 20:22:01.463459 19451 raft_consensus.cc:504] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.464071 19451 leader_election.cc:288] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [CANDIDATE]: Term 1 election: Requested vote from peers b6b8b7c5cee04854997b134b49cf3410 (127.14.127.194:36689), 72c142d050394960928d061a66207630 (127.14.127.195:46457)
I20260923 20:22:01.464497 19250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b5398a1ca4f65a8ebcbcb39f3dc2b" candidate_uuid: "eed9c09b0d76433ab67271e6d29f4f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b8b7c5cee04854997b134b49cf3410"
I20260923 20:22:01.464619 19382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b5398a1ca4f65a8ebcbcb39f3dc2b" candidate_uuid: "eed9c09b0d76433ab67271e6d29f4f68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c142d050394960928d061a66207630"
I20260923 20:22:01.464677 19250 raft_consensus.cc:3037] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:01.464834 19382 raft_consensus.cc:3037] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:01.466946 19250 raft_consensus.cc:2454] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed9c09b0d76433ab67271e6d29f4f68 in term 1.
I20260923 20:22:01.467157 19382 raft_consensus.cc:2454] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed9c09b0d76433ab67271e6d29f4f68 in term 1.
I20260923 20:22:01.467393 19050 leader_election.cc:302] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6b8b7c5cee04854997b134b49cf3410, eed9c09b0d76433ab67271e6d29f4f68; no voters:
I20260923 20:22:01.467708 19451 raft_consensus.cc:2781] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:01.468189 19451 raft_consensus.cc:686] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 1 LEADER]: Becoming Leader. State: Replica: eed9c09b0d76433ab67271e6d29f4f68, State: Running, Role: LEADER
I20260923 20:22:01.468755 19451 consensus_queue.cc:237] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } }
I20260923 20:22:01.473251 18972 catalog_manager.cc:5495] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed9c09b0d76433ab67271e6d29f4f68 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "eed9c09b0d76433ab67271e6d29f4f68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed9c09b0d76433ab67271e6d29f4f68" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 } health_report { overall_health: UNKNOWN } } }
I20260923 20:22:01.493772 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.488337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.14.127.193:44567)
0923 20:22:01.488598 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.488606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.488658 (+ 52us) server_negotiation.cc:407] Connection header received
0923 20:22:01.488910 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.488913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.488987 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.489137 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:01.489500 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.490201 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.491609 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.492010 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.492142 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.492628 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:01.493109 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:01.493307 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.493428 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.493507 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:22:01.501773 19382 raft_consensus.cc:1261] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 1 FOLLOWER]: Refusing update from remote peer eed9c09b0d76433ab67271e6d29f4f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:01.501775 19250 raft_consensus.cc:1261] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 1 FOLLOWER]: Refusing update from remote peer eed9c09b0d76433ab67271e6d29f4f68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:01.502866 19456 consensus_queue.cc:1035] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c142d050394960928d061a66207630" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:01.503221 19451 consensus_queue.cc:1035] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b8b7c5cee04854997b134b49cf3410" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 36689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:01.529304 18974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:01.529692 18974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:01.533370 18974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0f3c5c1a0504012aa4996fdaf82dc4a: Sending DeleteTablet for 3 replicas of tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b
I20260923 20:22:01.534715 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:01 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:55278
I20260923 20:22:01.534746 19230 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:01 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57140
I20260923 20:22:01.534793 19362 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:01 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57094
I20260923 20:22:01.539268 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 19035
I20260923 20:22:01.552809 19468 tablet_replica.cc:330] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: stopping tablet replica
I20260923 20:22:01.554351 19470 tablet_replica.cc:330] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: stopping tablet replica
I20260923 20:22:01.555226 19468 raft_consensus.cc:2229] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:01.555521 19470 raft_consensus.cc:2229] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:01.556185 19470 raft_consensus.cc:2258] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:01.557005 19468 raft_consensus.cc:2258] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:01.557307 19469 tablet_replica.cc:330] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: stopping tablet replica
I20260923 20:22:01.557646 19469 raft_consensus.cc:2229] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:01.558066 19469 raft_consensus.cc:2258] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 20:22:01 dist-test-slave-wmd0 krb5kdc[18930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:22:01.560241 19469 ts_tablet_manager.cc:1905] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:01.561609 19468 ts_tablet_manager.cc:1905] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:01.562080 19470 ts_tablet_manager.cc:1905] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:01.567553 19469 ts_tablet_manager.cc:1918] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:01.567792 19469 log.cc:1196] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/354b5398a1ca4f65a8ebcbcb39f3dc2b
I20260923 20:22:01.568389 19469 ts_tablet_manager.cc:1939] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P b6b8b7c5cee04854997b134b49cf3410: Deleting consensus metadata
I20260923 20:22:01.568840 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.540590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.14.127.193:44567)
0923 20:22:01.540818 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.540827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.543414 (+ 2587us) server_negotiation.cc:407] Connection header received
0923 20:22:01.543512 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.543520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.543622 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.543755 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:01.547504 (+ 3749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.548699 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.553096 (+ 4397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.553841 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.553940 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.560924 (+ 6984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:01.560940 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:01.560961 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:01.561029 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:01.565140 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.566054 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.566063 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.566067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.566251 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.566737 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.566744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.566752 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.567207 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:01.567555 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.568177 (+ 622us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.568441 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:22:01.570360 19157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:01.570866 18958 catalog_manager.cc:4841] TS b6b8b7c5cee04854997b134b49cf3410 (127.14.127.194:36689): tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b (table test-table [id=e6d744af48384b5389ac352089d2c2aa]) successfully deleted
I20260923 20:22:01.577237 19470 ts_tablet_manager.cc:1918] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:01.577446 19470 log.cc:1196] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/354b5398a1ca4f65a8ebcbcb39f3dc2b
I20260923 20:22:01.578039 19470 ts_tablet_manager.cc:1939] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P 72c142d050394960928d061a66207630: Deleting consensus metadata
I20260923 20:22:01.579684 18959 catalog_manager.cc:4841] TS 72c142d050394960928d061a66207630 (127.14.127.195:46457): tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b (table test-table [id=e6d744af48384b5389ac352089d2c2aa]) successfully deleted
I20260923 20:22:01.671800 19468 ts_tablet_manager.cc:1918] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:01.672109 19468 log.cc:1196] T 354b5398a1ca4f65a8ebcbcb39f3dc2b P eed9c09b0d76433ab67271e6d29f4f68: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/354b5398a1ca4f65a8ebcbcb39f3dc2b
I20260923 20:22:01.672434 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 19035
W20260923 20:22:01.689816 18960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.127.193:44567 (error 108)
I20260923 20:22:01.689922 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 19168
I20260923 20:22:01.691912 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.691374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44567 (local address 127.0.0.1:55304)
0923 20:22:01.691576 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.691719 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 20:22:01.692157 18960 catalog_manager.cc:4584] TS eed9c09b0d76433ab67271e6d29f4f68 (127.14.127.193:44567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 354b5398a1ca4f65a8ebcbcb39f3dc2b: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44567: connect: Connection refused (error 111)
Sep 23 20:22:01 dist-test-slave-wmd0 krb5kdc[18930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:22:01.706722 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.690713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.14.127.194:36689)
0923 20:22:01.690905 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.690909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.691087 (+ 178us) server_negotiation.cc:407] Connection header received
0923 20:22:01.691300 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.691306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.691390 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.691501 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:01.692921 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.693703 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.695121 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.695517 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.695671 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.699983 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:01.699994 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:01.700007 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:01.700039 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:01.703041 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.703873 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.703886 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.703892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.704060 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.704570 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.704580 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.704585 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.705210 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:01.705494 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.705968 (+ 474us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.706301 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:22:01.707465 19290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:01.745857 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.745308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44567 (local address 127.0.0.1:55318)
0923 20:22:01.745541 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.745653 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:22:01.746923 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.746498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44567 (local address 127.0.0.1:55320)
0923 20:22:01.746706 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.746792 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 20:22:01.798187 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 19168
I20260923 20:22:01.807665 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.807070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44567 (local address 127.0.0.1:55328)
0923 20:22:01.807334 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.807463 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 20:22:01.808769 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.808315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44567 (local address 127.0.0.1:55342)
0923 20:22:01.808506 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.808599 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 20:22:01.814988 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 19300
Sep 23 20:22:01 dist-test-slave-wmd0 krb5kdc[18930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:22:01.828744 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.815703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57096 (local address 127.14.127.195:46457)
0923 20:22:01.815857 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:01.815861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:01.815879 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:22:01.815954 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:01.815958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:01.816017 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:01.816105 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:01.817280 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.817990 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.819266 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:01.819685 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:01.819778 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:01.823916 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:01.823927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:01.823943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:01.823973 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:01.826461 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.827098 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.827104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.827106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.827220 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:01.827562 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:01.827565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:01.827567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:01.827900 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:01.828115 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:01.828334 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 20:22:01.828512 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:22:01.829438 19422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:01.889580 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.889004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44567 (local address 127.0.0.1:55356)
0923 20:22:01.889300 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.889394 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260923 20:22:01.890764 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:01.890289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44567 (local address 127.0.0.1:55364)
0923 20:22:01.890487 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:01.890596 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260923 20:22:01.919888 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 19300
I20260923 20:22:01.936599 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 18943
I20260923 20:22:01.937687 19004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:02.049036 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 18943
2026-09-23T20:22:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4154 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T20:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:02Z Disabled control of system clock
I20260923 20:22:02.114655 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:43809
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:43809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 20:22:02.427877 19480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:02.428158 19480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:02.428243 19480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:02.435315 19480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:02.435420 19480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:02.435456 19480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:02.435505 19480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:02.435549 19480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:02.444566 19480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:43809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:43809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:02.445768 19480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:02.447538 19480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:02.459432 19485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:02.459383 19488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:02.459339 19486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:02.460269 19480 server_base.cc:1034] running on GCE node
I20260923 20:22:02.461066 19480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:02.462375 19480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:02.463567 19480 hybrid_clock.cc:648] HybridClock initialized: now 1790194922463510 us; error 51 us; skew 500 ppm
I20260923 20:22:02.463908 19480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:02.477218 19480 webserver.cc:460] Webserver started at http://127.14.127.254:44145/ using document root <none> and password file <none>
I20260923 20:22:02.477955 19480 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:02.478084 19480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:02.478430 19480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:02.481492 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "7c597c436644488c8e61c7191b182e13"
format_stamp: "Formatted at 2026-09-23 20:22:02 on dist-test-slave-wmd0"
server_key: "cfe7bea6443f8fcd0220326ae38b8c22"
server_key_iv: "e1e258aaba573bc9d4c4a228b610500c"
server_key_version: "encryptionkey@0"
I20260923 20:22:02.482326 19480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "7c597c436644488c8e61c7191b182e13"
format_stamp: "Formatted at 2026-09-23 20:22:02 on dist-test-slave-wmd0"
server_key: "cfe7bea6443f8fcd0220326ae38b8c22"
server_key_iv: "e1e258aaba573bc9d4c4a228b610500c"
server_key_version: "encryptionkey@0"
I20260923 20:22:02.487376 19480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260923 20:22:02.490717 19494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:02.492069 19480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:22:02.492233 19480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "7c597c436644488c8e61c7191b182e13"
format_stamp: "Formatted at 2026-09-23 20:22:02 on dist-test-slave-wmd0"
server_key: "cfe7bea6443f8fcd0220326ae38b8c22"
server_key_iv: "e1e258aaba573bc9d4c4a228b610500c"
server_key_version: "encryptionkey@0"
I20260923 20:22:02.492455 19480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:02.498178 19480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:02.499032 19480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:02.499321 19480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:02.517235 19480 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:43809
I20260923 20:22:02.517266 19545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:43809 every 8 connection(s)
I20260923 20:22:02.518522 19480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:02.521826 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 19480
I20260923 20:22:02.521978 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:02.522241 19546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:02.522710 14847 external_mini_cluster.cc:1428] Setting key e5cd948c6e15a5e7280a1840c9a1a608
I20260923 20:22:02.531505 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:02.524761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.14.127.254:43809)
0923 20:22:02.525788 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:02.525799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:02.525839 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:22:02.527971 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:02.527993 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:02.528431 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:02.528770 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:02.529916 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:02.529922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:02.529935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:02.529950 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:02.530011 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:02.530035 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:02.530197 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:02.530331 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 20:22:02.530587 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":398,"threads_started":1}
I20260923 20:22:02.534747 19546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13: Bootstrap starting.
I20260923 20:22:02.537652 19546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:02.538686 19546 log.cc:824] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:02.541320 19546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13: No bootstrap required, opened a new log
I20260923 20:22:02.548394 19546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c597c436644488c8e61c7191b182e13" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43809 } }
I20260923 20:22:02.548833 19546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:02.548949 19546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c597c436644488c8e61c7191b182e13, State: Initialized, Role: FOLLOWER
I20260923 20:22:02.549480 19546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c597c436644488c8e61c7191b182e13" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43809 } }
I20260923 20:22:02.549706 19546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:02.549850 19546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:02.550017 19546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:02.552163 19546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c597c436644488c8e61c7191b182e13" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43809 } }
I20260923 20:22:02.552641 19546 leader_election.cc:302] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c597c436644488c8e61c7191b182e13; no voters:
I20260923 20:22:02.553200 19546 leader_election.cc:288] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:02.553299 19551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:02.554083 19551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [term 1 LEADER]: Becoming Leader. State: Replica: 7c597c436644488c8e61c7191b182e13, State: Running, Role: LEADER
I20260923 20:22:02.554596 19551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c597c436644488c8e61c7191b182e13" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43809 } }
I20260923 20:22:02.555334 19546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:02.557237 19553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c597c436644488c8e61c7191b182e13. Latest consensus state: current_term: 1 leader_uuid: "7c597c436644488c8e61c7191b182e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c597c436644488c8e61c7191b182e13" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43809 } } }
I20260923 20:22:02.557487 19552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c597c436644488c8e61c7191b182e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c597c436644488c8e61c7191b182e13" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43809 } } }
I20260923 20:22:02.557740 19553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:02.557812 19552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:02.559494 19556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:02.565083 19556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:02.573328 19556 catalog_manager.cc:1347] Generated new cluster ID: 398e9dcde9d74ae08de15b09d6a1b882
I20260923 20:22:02.573441 19556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:02.590246 19556 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:02.590747 19556 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:02.602218 19556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13: Generated new TSK 0
I20260923 20:22:02.603138 19556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:22:02.616155 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43809
--builtin_ntp_servers=127.14.127.212:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 20:22:02.934894 19570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:02.935158 19570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:02.935317 19570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:02.942392 19570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:02.942489 19570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:02.942644 19570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:02.951694 19570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:02.952926 19570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:02.954730 19570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:02.967803 19575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:02.967934 19576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:02.967940 19578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:02.968974 19570 server_base.cc:1034] running on GCE node
I20260923 20:22:02.969610 19570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:02.970530 19570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:02.971776 19570 hybrid_clock.cc:648] HybridClock initialized: now 1790194922971723 us; error 48 us; skew 500 ppm
I20260923 20:22:02.972101 19570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:02.974319 19570 webserver.cc:460] Webserver started at http://127.14.127.193:42333/ using document root <none> and password file <none>
I20260923 20:22:02.975075 19570 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:02.975204 19570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:02.975530 19570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:02.978256 19570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "f579ccb74b044ba1878911d842103bfb"
format_stamp: "Formatted at 2026-09-23 20:22:02 on dist-test-slave-wmd0"
server_key: "2c1620e6cd8f7d4f7ad1940b579e0646"
server_key_iv: "f84860d7d893ba3dc8b40a7c5ac55699"
server_key_version: "encryptionkey@0"
I20260923 20:22:02.979058 19570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "f579ccb74b044ba1878911d842103bfb"
format_stamp: "Formatted at 2026-09-23 20:22:02 on dist-test-slave-wmd0"
server_key: "2c1620e6cd8f7d4f7ad1940b579e0646"
server_key_iv: "f84860d7d893ba3dc8b40a7c5ac55699"
server_key_version: "encryptionkey@0"
I20260923 20:22:02.983903 19570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 20:22:02.987721 19584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:02.988986 19570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:02.989166 19570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "f579ccb74b044ba1878911d842103bfb"
format_stamp: "Formatted at 2026-09-23 20:22:02 on dist-test-slave-wmd0"
server_key: "2c1620e6cd8f7d4f7ad1940b579e0646"
server_key_iv: "f84860d7d893ba3dc8b40a7c5ac55699"
server_key_version: "encryptionkey@0"
I20260923 20:22:02.989382 19570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:02.994864 19570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:02.995731 19570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:02.996003 19570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:02.996970 19570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:02.998317 19570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:02.998425 19570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:02.998561 19570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:02.998682 19570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 20:22:03.031569 19570 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:37005
I20260923 20:22:03.031605 19696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:37005 every 8 connection(s)
I20260923 20:22:03.032934 19570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:03.036019 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 19570
I20260923 20:22:03.036209 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:03.037073 14847 external_mini_cluster.cc:1428] Setting key 063c0acce7a5576550fbbe217db42c6c
I20260923 20:22:03.040184 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43809
--builtin_ntp_servers=127.14.127.212:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 20:22:03.041925 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.035469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:59103 (local address 127.14.127.254:43809)
0923 20:22:03.035957 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:03.035963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:03.037494 (+ 1531us) server_negotiation.cc:407] Connection header received
0923 20:22:03.039599 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:03.039602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:03.039651 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:03.039740 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:03.041237 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:03.041241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:03.041245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:03.041259 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:03.041298 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:03.041309 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:03.041403 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:03.041647 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:22:03.041694 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":257,"threads_started":1}
I20260923 20:22:03.043258 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.036104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43809 (local address 127.14.127.193:59103)
0923 20:22:03.037310 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:03.037363 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:03.039306 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:03.040031 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:03.040046 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:03.040178 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:03.040872 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:03.040878 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:03.040931 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:03.041060 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:03.041411 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:03.041423 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:03.041436 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 20:22:03.041725 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 20:22:03.041985 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":344,"threads_started":1}
I20260923 20:22:03.044831 19697 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43809
I20260923 20:22:03.045133 19697 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:03.045766 19697 heartbeater.cc:511] Master 127.14.127.254:43809 requested a full tablet report, sending...
I20260923 20:22:03.047899 19511 ts_manager.cc:194] Registered new tserver with Master: f579ccb74b044ba1878911d842103bfb (127.14.127.193:37005)
W20260923 20:22:03.354651 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:03.354956 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:03.355122 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:03.362313 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:03.362409 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:03.362650 19701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:03.371899 19701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:03.373217 19701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:03.375044 19701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:03.388082 19707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:03.388043 19706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:03.388405 19709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:03.389217 19701 server_base.cc:1034] running on GCE node
I20260923 20:22:03.389873 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:03.390822 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:03.392041 19701 hybrid_clock.cc:648] HybridClock initialized: now 1790194923391984 us; error 36 us; skew 500 ppm
I20260923 20:22:03.392362 19701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:03.394767 19701 webserver.cc:460] Webserver started at http://127.14.127.194:33841/ using document root <none> and password file <none>
I20260923 20:22:03.395483 19701 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:03.395597 19701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:03.395977 19701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:03.398972 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04"
format_stamp: "Formatted at 2026-09-23 20:22:03 on dist-test-slave-wmd0"
server_key: "7ed7bfc2a75ee8690012ee6217a848aa"
server_key_iv: "372a1737dbebc1ac5428bc8cd1d8c223"
server_key_version: "encryptionkey@0"
I20260923 20:22:03.399801 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04"
format_stamp: "Formatted at 2026-09-23 20:22:03 on dist-test-slave-wmd0"
server_key: "7ed7bfc2a75ee8690012ee6217a848aa"
server_key_iv: "372a1737dbebc1ac5428bc8cd1d8c223"
server_key_version: "encryptionkey@0"
I20260923 20:22:03.404865 19701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:22:03.408308 19715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:03.409695 19701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:22:03.409852 19701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04"
format_stamp: "Formatted at 2026-09-23 20:22:03 on dist-test-slave-wmd0"
server_key: "7ed7bfc2a75ee8690012ee6217a848aa"
server_key_iv: "372a1737dbebc1ac5428bc8cd1d8c223"
server_key_version: "encryptionkey@0"
I20260923 20:22:03.410072 19701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:03.415845 19701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:03.416621 19701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:03.416886 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:03.417773 19701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:03.419180 19701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:03.419288 19701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:03.419414 19701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:03.419508 19701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:03.452118 19701 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:37235
I20260923 20:22:03.452171 19827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:37235 every 8 connection(s)
I20260923 20:22:03.453500 19701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:03.460880 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 19701
I20260923 20:22:03.461084 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:03.461722 14847 external_mini_cluster.cc:1428] Setting key 54fd95e88d74c2432a38c4483d826280
I20260923 20:22:03.461927 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.456191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:54745 (local address 127.14.127.254:43809)
0923 20:22:03.456432 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:03.456438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:03.457845 (+ 1407us) server_negotiation.cc:407] Connection header received
0923 20:22:03.459537 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:03.459541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:03.459595 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:03.459682 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:03.461190 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:03.461197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:03.461202 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:03.461229 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:03.461263 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:03.461275 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:03.461371 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:03.461600 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:22:03.461651 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 20:22:03.462980 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.456791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43809 (local address 127.14.127.194:54745)
0923 20:22:03.457682 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:03.457718 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:03.459295 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:03.459944 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:03.459952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:03.460054 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:03.460732 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:03.460735 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:03.460786 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:03.460908 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:03.461376 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:03.461386 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:03.461394 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:22:03.461680 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 20:22:03.461951 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":265,"threads_started":1}
I20260923 20:22:03.464349 19828 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43809
I20260923 20:22:03.464670 19828 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:03.464738 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43809
--builtin_ntp_servers=127.14.127.212:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 20:22:03.465346 19828 heartbeater.cc:511] Master 127.14.127.254:43809 requested a full tablet report, sending...
I20260923 20:22:03.466905 19511 ts_manager.cc:194] Registered new tserver with Master: 9c1b276dc36c4dfa8d84b74e41b3bd04 (127.14.127.194:37235)
W20260923 20:22:03.787201 19831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:03.787490 19831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:03.787662 19831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:03.795001 19831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:03.795123 19831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:03.795310 19831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:03.804291 19831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:03.805578 19831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:03.807343 19831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:03.820623 19836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:03.820677 19839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:03.820699 19837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:03.821614 19831 server_base.cc:1034] running on GCE node
I20260923 20:22:03.822209 19831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:03.823635 19831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:03.824867 19831 hybrid_clock.cc:648] HybridClock initialized: now 1790194923824814 us; error 63 us; skew 500 ppm
I20260923 20:22:03.825191 19831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:03.827527 19831 webserver.cc:460] Webserver started at http://127.14.127.195:38601/ using document root <none> and password file <none>
I20260923 20:22:03.828217 19831 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:03.828349 19831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:03.828679 19831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:03.831514 19831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "da21cced2dd4463d900f9e05cc612fb7"
format_stamp: "Formatted at 2026-09-23 20:22:03 on dist-test-slave-wmd0"
server_key: "ae0b07d41c8f9d7db50eec4d249f566f"
server_key_iv: "1650454b03eeef5db08f5419c5a5df86"
server_key_version: "encryptionkey@0"
I20260923 20:22:03.832340 19831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "da21cced2dd4463d900f9e05cc612fb7"
format_stamp: "Formatted at 2026-09-23 20:22:03 on dist-test-slave-wmd0"
server_key: "ae0b07d41c8f9d7db50eec4d249f566f"
server_key_iv: "1650454b03eeef5db08f5419c5a5df86"
server_key_version: "encryptionkey@0"
I20260923 20:22:03.837502 19831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:22:03.840909 19845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:03.842155 19831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:22:03.842353 19831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "da21cced2dd4463d900f9e05cc612fb7"
format_stamp: "Formatted at 2026-09-23 20:22:03 on dist-test-slave-wmd0"
server_key: "ae0b07d41c8f9d7db50eec4d249f566f"
server_key_iv: "1650454b03eeef5db08f5419c5a5df86"
server_key_version: "encryptionkey@0"
I20260923 20:22:03.842576 19831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:03.847980 19831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:03.848724 19831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:03.849000 19831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:03.849862 19831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:03.851248 19831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:03.851367 19831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:03.851502 19831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:03.851588 19831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:03.884652 19831 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:41329
I20260923 20:22:03.884702 19957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:41329 every 8 connection(s)
I20260923 20:22:03.885949 19831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:03.893872 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.888580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:57627 (local address 127.14.127.254:43809)
0923 20:22:03.888810 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:03.888817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:03.890192 (+ 1375us) server_negotiation.cc:407] Connection header received
0923 20:22:03.891864 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:03.891868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:03.891922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:03.892009 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:03.893240 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:03.893245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:03.893248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:03.893264 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:03.893294 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:03.893305 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:03.893413 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:03.893563 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 20:22:03.893640 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:22:03.894696 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 19831
I20260923 20:22:03.894868 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:03.895143 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.889117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43809 (local address 127.14.127.195:57627)
0923 20:22:03.890037 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:03.890072 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:03.891618 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:03.892210 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:03.892218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:03.892305 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:03.892941 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:03.892944 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:03.892987 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:03.893079 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:03.893390 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:03.893396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:03.893401 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:22:03.893609 (+ 208us) client_negotiation.cc:240] Negotiation successful
0923 20:22:03.893860 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":276,"threads_started":1}
I20260923 20:22:03.895416 14847 external_mini_cluster.cc:1428] Setting key 84212dfe36a5b7579f24c6670eb57c45
I20260923 20:22:03.896636 19958 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43809
I20260923 20:22:03.896996 19958 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:03.897701 19958 heartbeater.cc:511] Master 127.14.127.254:43809 requested a full tablet report, sending...
I20260923 20:22:03.899125 19511 ts_manager.cc:194] Registered new tserver with Master: da21cced2dd4463d900f9e05cc612fb7 (127.14.127.195:41329)
I20260923 20:22:03.910470 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:22:03.917690 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.914956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.14.127.254:43809)
0923 20:22:03.915125 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:03.915129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:03.915675 (+ 546us) server_negotiation.cc:407] Connection header received
0923 20:22:03.915887 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:03.915894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:03.915970 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:03.916076 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:03.917038 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:03.917044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:03.917049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:03.917067 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:03.917107 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:03.917118 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:03.917242 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:03.917347 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 20:22:03.917416 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:22:03.921555 19511 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:03.928272 19511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:03.962014 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.939820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.14.127.193:37005)
0923 20:22:03.958813 (+ 18993us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:03.958823 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:03.958852 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:22:03.958921 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:03.958930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:03.959178 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:03.960147 (+ 969us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:03.961092 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:03.961101 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:03.961115 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:03.961132 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:03.961191 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:03.961217 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:03.961363 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:03.961572 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 20:22:03.961692 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18838,"thread_start_us":18688,"threads_started":1}
I20260923 20:22:03.962126 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.939656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37005 (local address 127.0.0.1:42062)
0923 20:22:03.954307 (+ 14651us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:03.954379 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:03.954582 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:03.959500 (+ 4918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:03.959511 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:03.959610 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:03.960544 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:03.960548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:03.960622 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:03.960718 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:03.961345 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:03.961357 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:03.961365 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:22:03.961566 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 20:22:03.961672 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14282,"thread_start_us":14967,"threads_started":1}
I20260923 20:22:03.962293 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.940928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.14.127.194:37235)
0923 20:22:03.957355 (+ 16427us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:03.957368 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:03.958766 (+ 1398us) server_negotiation.cc:407] Connection header received
0923 20:22:03.958843 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:03.958850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:03.959048 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:03.959158 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:03.961122 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:03.961130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:03.961141 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:03.961158 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:03.961208 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:03.961231 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:03.961395 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:03.961860 (+ 465us) server_negotiation.cc:299] Negotiation successful
0923 20:22:03.961950 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16209,"thread_start_us":16068,"threads_started":1}
I20260923 20:22:03.962723 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.940300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:37235 (local address 127.0.0.1:52222)
0923 20:22:03.958420 (+ 18120us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:03.958443 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:03.958557 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:03.959911 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:03.959917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:03.959947 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:03.960885 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:03.960887 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:03.960898 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:03.960941 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:03.961437 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:03.961442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:03.961447 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:22:03.961645 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 20:22:03.961695 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18000,"thread_start_us":17231,"threads_started":1}
I20260923 20:22:03.962757 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.940655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.14.127.195:41329)
0923 20:22:03.958404 (+ 17749us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:03.958417 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:03.958464 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:22:03.958553 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:03.958560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:03.958871 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:03.959059 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:03.961151 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:03.961158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:03.961167 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:03.961182 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:03.961239 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:03.961253 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:03.961448 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:03.961657 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 20:22:03.961751 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17534,"thread_start_us":17340,"threads_started":1}
I20260923 20:22:03.963625 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:03.939933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:41329 (local address 127.0.0.1:36586)
0923 20:22:03.957844 (+ 17911us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:03.957866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:03.957982 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:03.960225 (+ 2243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:03.960232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:03.960259 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:03.960869 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:03.960873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:03.960890 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:03.960941 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:03.961502 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:03.961506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:03.961513 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 20:22:03.962748 (+ 1235us) client_negotiation.cc:240] Negotiation successful
0923 20:22:03.963191 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17739,"thread_start_us":18292,"threads_started":1}
I20260923 20:22:03.966741 19632 tablet_service.cc:1467] Processing CreateTablet for tablet 0f674f6022df43589b79fbdfb32df50c (DEFAULT_TABLE table=test-table [id=2f9827ae6a9a4f28809f2cc80f90c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:03.966741 19763 tablet_service.cc:1467] Processing CreateTablet for tablet 0f674f6022df43589b79fbdfb32df50c (DEFAULT_TABLE table=test-table [id=2f9827ae6a9a4f28809f2cc80f90c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:03.968271 19763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f674f6022df43589b79fbdfb32df50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:03.968286 19632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f674f6022df43589b79fbdfb32df50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:03.970084 19893 tablet_service.cc:1467] Processing CreateTablet for tablet 0f674f6022df43589b79fbdfb32df50c (DEFAULT_TABLE table=test-table [id=2f9827ae6a9a4f28809f2cc80f90c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:03.971298 19893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f674f6022df43589b79fbdfb32df50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:03.982173 19976 tablet_bootstrap.cc:492] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: Bootstrap starting.
I20260923 20:22:03.985713 19976 tablet_bootstrap.cc:654] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:03.987149 19976 log.cc:824] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:03.989055 19978 tablet_bootstrap.cc:492] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Bootstrap starting.
I20260923 20:22:03.990151 19977 tablet_bootstrap.cc:492] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: Bootstrap starting.
I20260923 20:22:03.990303 19976 tablet_bootstrap.cc:492] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: No bootstrap required, opened a new log
I20260923 20:22:03.990574 19976 ts_tablet_manager.cc:1397] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 20:22:03.992951 19978 tablet_bootstrap.cc:654] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:03.993731 19977 tablet_bootstrap.cc:654] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:03.994266 19978 log.cc:824] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:03.995048 19977 log.cc:824] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:03.998414 19976 raft_consensus.cc:348] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:03.998862 19976 raft_consensus.cc:374] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:03.998968 19976 raft_consensus.cc:729] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da21cced2dd4463d900f9e05cc612fb7, State: Initialized, Role: FOLLOWER
I20260923 20:22:03.999652 19976 consensus_queue.cc:260] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:04.000811 19958 heartbeater.cc:503] Master 127.14.127.254:43809 was elected leader, sending a full tablet report...
I20260923 20:22:04.001740 19978 tablet_bootstrap.cc:492] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: No bootstrap required, opened a new log
I20260923 20:22:04.002014 19978 ts_tablet_manager.cc:1397] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260923 20:22:04.002470 19977 tablet_bootstrap.cc:492] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: No bootstrap required, opened a new log
I20260923 20:22:04.002764 19977 ts_tablet_manager.cc:1397] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260923 20:22:04.002764 19976 ts_tablet_manager.cc:1428] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260923 20:22:04.013006 19978 raft_consensus.cc:348] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:04.013521 19978 raft_consensus.cc:374] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:04.013625 19978 raft_consensus.cc:729] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f579ccb74b044ba1878911d842103bfb, State: Initialized, Role: FOLLOWER
I20260923 20:22:04.013494 19977 raft_consensus.cc:348] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:04.013973 19977 raft_consensus.cc:374] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:04.014071 19977 raft_consensus.cc:729] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1b276dc36c4dfa8d84b74e41b3bd04, State: Initialized, Role: FOLLOWER
I20260923 20:22:04.014375 19978 consensus_queue.cc:260] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:04.014694 19977 consensus_queue.cc:260] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:04.015614 19697 heartbeater.cc:503] Master 127.14.127.254:43809 was elected leader, sending a full tablet report...
I20260923 20:22:04.016292 19978 ts_tablet_manager.cc:1428] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260923 20:22:04.020427 19828 heartbeater.cc:503] Master 127.14.127.254:43809 was elected leader, sending a full tablet report...
I20260923 20:22:04.020964 19977 ts_tablet_manager.cc:1428] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20260923 20:22:04.035291 19698 tablet.cc:2369] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:22:04.046603 19983 raft_consensus.cc:482] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:04.047001 19983 raft_consensus.cc:504] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:04.048712 19983 leader_election.cc:288] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da21cced2dd4463d900f9e05cc612fb7 (127.14.127.195:41329), 9c1b276dc36c4dfa8d84b74e41b3bd04 (127.14.127.194:37235)
I20260923 20:22:04.052205 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.049011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:56337 (local address 127.14.127.195:41329)
0923 20:22:04.049235 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:04.049242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:04.050126 (+ 884us) server_negotiation.cc:407] Connection header received
0923 20:22:04.050233 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:04.050239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:04.050305 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:04.050409 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:04.051528 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:04.051534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:04.051538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:04.051558 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:04.051598 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:04.051616 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:04.051739 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:04.051834 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 20:22:04.051910 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:22:04.052383 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.049277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:59261 (local address 127.14.127.194:37235)
0923 20:22:04.049493 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:04.049501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:04.050268 (+ 767us) server_negotiation.cc:407] Connection header received
0923 20:22:04.050402 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:04.050408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:04.050464 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:04.050570 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:04.051731 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:04.051736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:04.051740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:04.051761 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:04.051804 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:04.051822 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:04.051954 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:04.052075 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 20:22:04.052139 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:22:04.052685 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.049182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:37235 (local address 127.14.127.193:59261)
0923 20:22:04.050130 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.050150 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:04.050258 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:04.050602 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:04.050605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:04.050671 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:04.051491 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:04.051494 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:04.051510 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:04.051552 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:04.051939 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:04.051944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:04.051949 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:22:04.052124 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 20:22:04.052173 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":532,"threads_started":1}
I20260923 20:22:04.052891 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.049164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:41329 (local address 127.14.127.193:56337)
0923 20:22:04.050010 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.050026 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:04.050132 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:04.050450 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:04.050455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:04.050479 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:04.051268 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:04.051271 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:04.051284 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:04.051322 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:04.051721 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:04.051725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:04.051730 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:22:04.051899 (+ 169us) client_negotiation.cc:240] Negotiation successful
0923 20:22:04.051956 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":469,"threads_started":1}
I20260923 20:22:04.053694 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f674f6022df43589b79fbdfb32df50c" candidate_uuid: "f579ccb74b044ba1878911d842103bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" is_pre_election: true
I20260923 20:22:04.053905 19913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f674f6022df43589b79fbdfb32df50c" candidate_uuid: "f579ccb74b044ba1878911d842103bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da21cced2dd4463d900f9e05cc612fb7" is_pre_election: true
I20260923 20:22:04.054283 19783 raft_consensus.cc:2454] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f579ccb74b044ba1878911d842103bfb in term 0.
I20260923 20:22:04.054394 19913 raft_consensus.cc:2454] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f579ccb74b044ba1878911d842103bfb in term 0.
I20260923 20:22:04.055054 19588 leader_election.cc:302] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da21cced2dd4463d900f9e05cc612fb7, f579ccb74b044ba1878911d842103bfb; no voters:
I20260923 20:22:04.055490 19983 raft_consensus.cc:2781] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:04.055641 19983 raft_consensus.cc:482] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:04.055740 19983 raft_consensus.cc:3037] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:04.058000 19983 raft_consensus.cc:504] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:04.058571 19983 leader_election.cc:288] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [CANDIDATE]: Term 1 election: Requested vote from peers da21cced2dd4463d900f9e05cc612fb7 (127.14.127.195:41329), 9c1b276dc36c4dfa8d84b74e41b3bd04 (127.14.127.194:37235)
I20260923 20:22:04.059078 19913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f674f6022df43589b79fbdfb32df50c" candidate_uuid: "f579ccb74b044ba1878911d842103bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da21cced2dd4463d900f9e05cc612fb7"
I20260923 20:22:04.059175 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f674f6022df43589b79fbdfb32df50c" candidate_uuid: "f579ccb74b044ba1878911d842103bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04"
I20260923 20:22:04.059274 19913 raft_consensus.cc:3037] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:04.059419 19783 raft_consensus.cc:3037] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:04.061769 19913 raft_consensus.cc:2454] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f579ccb74b044ba1878911d842103bfb in term 1.
I20260923 20:22:04.061858 19783 raft_consensus.cc:2454] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f579ccb74b044ba1878911d842103bfb in term 1.
I20260923 20:22:04.062245 19588 leader_election.cc:302] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9c1b276dc36c4dfa8d84b74e41b3bd04, da21cced2dd4463d900f9e05cc612fb7, f579ccb74b044ba1878911d842103bfb; no voters:
I20260923 20:22:04.062577 19983 raft_consensus.cc:2781] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:04.063050 19983 raft_consensus.cc:686] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 1 LEADER]: Becoming Leader. State: Replica: f579ccb74b044ba1878911d842103bfb, State: Running, Role: LEADER
I20260923 20:22:04.063582 19983 consensus_queue.cc:237] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } }
I20260923 20:22:04.067938 19511 catalog_manager.cc:5495] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f579ccb74b044ba1878911d842103bfb (127.14.127.193). New cstate: current_term: 1 leader_uuid: "f579ccb74b044ba1878911d842103bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f579ccb74b044ba1878911d842103bfb" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 } health_report { overall_health: UNKNOWN } } }
I20260923 20:22:04.089043 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.086816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.14.127.193:37005)
0923 20:22:04.087001 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:04.087007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:04.087056 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:22:04.087345 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:04.087350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:04.087432 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:04.087551 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:04.088421 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:04.088426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:04.088431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:04.088454 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:04.088486 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:04.088497 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:04.088618 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:04.088708 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 20:22:04.088768 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:22:04.097453 19783 raft_consensus.cc:1261] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 1 FOLLOWER]: Refusing update from remote peer f579ccb74b044ba1878911d842103bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:04.097527 19913 raft_consensus.cc:1261] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 1 FOLLOWER]: Refusing update from remote peer f579ccb74b044ba1878911d842103bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:04.098471 19987 consensus_queue.cc:1035] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1b276dc36c4dfa8d84b74e41b3bd04" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:04.098886 19983 consensus_queue.cc:1035] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "da21cced2dd4463d900f9e05cc612fb7" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:04.108409 19990 mvcc.cc:205] Tried to move back new op lower bound from 7332638409093214208 to 7332638408968376320. Current Snapshot: MvccSnapshot[applied={T|T < 7332638409093214208}]
I20260923 20:22:04.111727 19991 mvcc.cc:205] Tried to move back new op lower bound from 7332638409093214208 to 7332638408968376320. Current Snapshot: MvccSnapshot[applied={T|T < 7332638409093214208}]
I20260923 20:22:04.113596 19992 mvcc.cc:205] Tried to move back new op lower bound from 7332638409093214208 to 7332638408968376320. Current Snapshot: MvccSnapshot[applied={T|T < 7332638409093214208}]
I20260923 20:22:04.123883 19511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:04.124179 19511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:04.127707 19511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c597c436644488c8e61c7191b182e13: Sending DeleteTablet for 3 replicas of tablet 0f674f6022df43589b79fbdfb32df50c
I20260923 20:22:04.128731 19632 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f674f6022df43589b79fbdfb32df50c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:04 UTC) from {username='slave'} at 127.0.0.1:42062
I20260923 20:22:04.129137 19763 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f674f6022df43589b79fbdfb32df50c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:04 UTC) from {username='slave'} at 127.0.0.1:52222
I20260923 20:22:04.129158 19893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f674f6022df43589b79fbdfb32df50c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:04 UTC) from {username='slave'} at 127.0.0.1:36586
I20260923 20:22:04.131925 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 19570
I20260923 20:22:04.140188 20002 tablet_replica.cc:330] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: stopping tablet replica
I20260923 20:22:04.140309 20001 tablet_replica.cc:330] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: stopping tablet replica
I20260923 20:22:04.140650 20001 raft_consensus.cc:2229] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:04.140647 20002 raft_consensus.cc:2229] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:04.141148 20002 raft_consensus.cc:2258] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:04.141148 20001 raft_consensus.cc:2258] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:04.141919 20000 tablet_replica.cc:330] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: stopping tablet replica
I20260923 20:22:04.142338 20000 raft_consensus.cc:2229] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:04.143005 20000 raft_consensus.cc:2258] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:04.143814 20001 ts_tablet_manager.cc:1905] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:04.146095 20002 ts_tablet_manager.cc:1905] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:04.145845 20000 ts_tablet_manager.cc:1905] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:04.150978 20001 ts_tablet_manager.cc:1918] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:04.151165 20001 log.cc:1196] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/0f674f6022df43589b79fbdfb32df50c
I20260923 20:22:04.151688 20001 ts_tablet_manager.cc:1939] T 0f674f6022df43589b79fbdfb32df50c P 9c1b276dc36c4dfa8d84b74e41b3bd04: Deleting consensus metadata
I20260923 20:22:04.153322 20002 ts_tablet_manager.cc:1918] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:04.153574 20002 log.cc:1196] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/0f674f6022df43589b79fbdfb32df50c
I20260923 20:22:04.153573 19495 catalog_manager.cc:4841] TS 9c1b276dc36c4dfa8d84b74e41b3bd04 (127.14.127.194:37235): tablet 0f674f6022df43589b79fbdfb32df50c (table test-table [id=2f9827ae6a9a4f28809f2cc80f90c420]) successfully deleted
I20260923 20:22:04.154551 20002 ts_tablet_manager.cc:1939] T 0f674f6022df43589b79fbdfb32df50c P da21cced2dd4463d900f9e05cc612fb7: Deleting consensus metadata
I20260923 20:22:04.154901 20000 ts_tablet_manager.cc:1918] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:04.155092 20000 log.cc:1196] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/0f674f6022df43589b79fbdfb32df50c
I20260923 20:22:04.155299 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.133495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.14.127.193:37005)
0923 20:22:04.133707 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:04.133716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:04.146074 (+ 12358us) server_negotiation.cc:407] Connection header received
0923 20:22:04.146314 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:04.146319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:04.146387 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:04.146496 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:04.154356 (+ 7860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:04.154362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:04.154368 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:04.154390 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:04.154433 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:04.154449 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:04.154577 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:04.154857 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:22:04.154928 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:22:04.155663 20000 ts_tablet_manager.cc:1939] T 0f674f6022df43589b79fbdfb32df50c P f579ccb74b044ba1878911d842103bfb: Deleting consensus metadata
I20260923 20:22:04.156266 19692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:04.156873 19498 catalog_manager.cc:4841] TS da21cced2dd4463d900f9e05cc612fb7 (127.14.127.195:41329): tablet 0f674f6022df43589b79fbdfb32df50c (table test-table [id=2f9827ae6a9a4f28809f2cc80f90c420]) successfully deleted
I20260923 20:22:04.249353 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 19570
I20260923 20:22:04.267241 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 19701
W20260923 20:22:04.267210 19497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.127.193:37005 (error 108)
I20260923 20:22:04.269209 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.268762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37005 (local address 127.0.0.1:42078)
0923 20:22:04.268962 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.269076 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 20:22:04.269467 19497 catalog_manager.cc:4584] TS f579ccb74b044ba1878911d842103bfb (127.14.127.193:37005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f674f6022df43589b79fbdfb32df50c: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37005: connect: Connection refused (error 111)
I20260923 20:22:04.270342 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.268138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.14.127.194:37235)
0923 20:22:04.268337 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:04.268344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:04.268367 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:22:04.268556 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:04.268562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:04.268662 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:04.268750 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:04.269807 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:04.269811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:04.269814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:04.269829 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:04.269859 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:04.269870 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:04.269968 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:04.270096 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 20:22:04.270144 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:22:04.271036 19823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:04.323398 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.322929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37005 (local address 127.0.0.1:42088)
0923 20:22:04.323139 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.323225 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:22:04.324376 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.324035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37005 (local address 127.0.0.1:42102)
0923 20:22:04.324177 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.324257 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:22:04.357121 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 19701
I20260923 20:22:04.373844 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 19831
I20260923 20:22:04.376426 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.374585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.14.127.195:41329)
0923 20:22:04.374777 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:04.374781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:04.374820 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:22:04.374986 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:04.374990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:04.375067 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:04.375153 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:04.375934 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:04.375938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:04.375942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:04.375956 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:04.375984 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:04.375996 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:04.376097 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:04.376157 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 20:22:04.376200 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:22:04.377033 19953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:04.384814 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.384261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37005 (local address 127.0.0.1:42114)
0923 20:22:04.384497 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.384622 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 20:22:04.385857 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.385455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37005 (local address 127.0.0.1:42124)
0923 20:22:04.385614 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.385703 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 20:22:04.465739 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 19831
I20260923 20:22:04.467240 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.465884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37005 (local address 127.0.0.1:42134)
0923 20:22:04.466919 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.467047 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":869}
I20260923 20:22:04.468364 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:04.467974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37005 (local address 127.0.0.1:42142)
0923 20:22:04.468191 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:04.468260 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 20:22:04.482249 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 19480
I20260923 20:22:04.483419 19541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:04.588575 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 19480
2026-09-23T20:22:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2538 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 20:22:04.630546 14847 mini_oidc.cc:152] Starting JWKS server
I20260923 20:22:04.634473 14847 webserver.cc:460] Webserver started at https://127.0.0.1:38135/ using document root <none> and password file <none>
I20260923 20:22:04.634866 14847 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:22:04.635537 14847 webserver.cc:460] Webserver started at http://127.0.0.1:45737/ using document root <none> and password file <none>
2026-09-23T20:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:04Z Disabled control of system clock
I20260923 20:22:04.661520 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36055
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:36055
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38135/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/testchainca.cert with env {}
W20260923 20:22:04.983213 20014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:04.983554 20014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:04.983718 20014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:04.990945 20014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:04.991094 20014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:04.991186 20014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:04.991297 20014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:04.991389 20014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:22:04.991479 20014 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:22:04.991559 20014 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38135/jwks.json
I20260923 20:22:05.001443 20014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:36055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38135/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:05.002796 20014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:05.004138 20014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:05.015830 20022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:05.016042 20020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:05.016417 20019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:05.016700 20014 server_base.cc:1034] running on GCE node
I20260923 20:22:05.017434 20014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:05.018909 20014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:05.020121 20014 hybrid_clock.cc:648] HybridClock initialized: now 1790194925020060 us; error 54 us; skew 500 ppm
I20260923 20:22:05.020509 20014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:05.022794 20014 webserver.cc:460] Webserver started at http://127.14.127.254:46105/ using document root <none> and password file <none>
I20260923 20:22:05.023474 20014 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:05.023594 20014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:05.023922 20014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:05.026819 20014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "6cb0bd1fad594ba486f975783049e31d"
format_stamp: "Formatted at 2026-09-23 20:22:05 on dist-test-slave-wmd0"
server_key: "3ed0f2dfad93930ff46acc8366ba4cbf"
server_key_iv: "018fec4912136d3b3dcbda1888b94a3b"
server_key_version: "encryptionkey@0"
I20260923 20:22:05.027706 20014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "6cb0bd1fad594ba486f975783049e31d"
format_stamp: "Formatted at 2026-09-23 20:22:05 on dist-test-slave-wmd0"
server_key: "3ed0f2dfad93930ff46acc8366ba4cbf"
server_key_iv: "018fec4912136d3b3dcbda1888b94a3b"
server_key_version: "encryptionkey@0"
I20260923 20:22:05.032743 20014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 20:22:05.036245 20028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:05.037626 20014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 20:22:05.037799 20014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "6cb0bd1fad594ba486f975783049e31d"
format_stamp: "Formatted at 2026-09-23 20:22:05 on dist-test-slave-wmd0"
server_key: "3ed0f2dfad93930ff46acc8366ba4cbf"
server_key_iv: "018fec4912136d3b3dcbda1888b94a3b"
server_key_version: "encryptionkey@0"
I20260923 20:22:05.038022 20014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:05.053074 20014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:05.053875 20014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:05.054193 20014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:05.072533 20014 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:36055
I20260923 20:22:05.072568 20079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:36055 every 8 connection(s)
I20260923 20:22:05.073865 20014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:05.077446 20080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:05.079902 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20014
I20260923 20:22:05.080122 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:05.080721 14847 external_mini_cluster.cc:1428] Setting key 14fad8f587b9b925de40e6a94c906695
I20260923 20:22:05.090426 20080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d: Bootstrap starting.
I20260923 20:22:05.094111 20080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:05.095469 20080 log.cc:824] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:05.096202 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:05.082658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.14.127.254:36055)
0923 20:22:05.083660 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:05.083674 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:05.083722 (+ 48us) server_negotiation.cc:407] Connection header received
0923 20:22:05.085982 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:05.086010 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:05.086669 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:05.087103 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:05.087676 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.089642 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.091048 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.092062 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.092254 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:05.093300 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:05.093311 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:05.093327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:05.094018 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:05.094137 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:05.094163 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:05.094290 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:05.094529 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 20:22:05.094875 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":286,"threads_started":1}
I20260923 20:22:05.098575 20080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d: No bootstrap required, opened a new log
I20260923 20:22:05.100031 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:22:05.106144 20080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb0bd1fad594ba486f975783049e31d" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36055 } }
I20260923 20:22:05.106557 20080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:05.106722 20080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb0bd1fad594ba486f975783049e31d, State: Initialized, Role: FOLLOWER
I20260923 20:22:05.107363 20080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb0bd1fad594ba486f975783049e31d" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36055 } }
I20260923 20:22:05.107590 20080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:05.107714 20080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:05.107884 20080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:05.109982 20080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb0bd1fad594ba486f975783049e31d" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36055 } }
I20260923 20:22:05.110476 20080 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb0bd1fad594ba486f975783049e31d; no voters:
I20260923 20:22:05.111047 20080 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:05.111202 20085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:05.111939 20085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [term 1 LEADER]: Becoming Leader. State: Replica: 6cb0bd1fad594ba486f975783049e31d, State: Running, Role: LEADER
I20260923 20:22:05.112476 20085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb0bd1fad594ba486f975783049e31d" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36055 } }
I20260923 20:22:05.113206 20080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:05.115468 20086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb0bd1fad594ba486f975783049e31d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb0bd1fad594ba486f975783049e31d" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36055 } } }
I20260923 20:22:05.115926 20086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:05.116503 20087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb0bd1fad594ba486f975783049e31d. Latest consensus state: current_term: 1 leader_uuid: "6cb0bd1fad594ba486f975783049e31d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb0bd1fad594ba486f975783049e31d" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36055 } } }
I20260923 20:22:05.116868 20087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:05.118192 20094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:05.123683 20094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:05.132849 20094 catalog_manager.cc:1347] Generated new cluster ID: 545f02f80e894920b478002f22784aab
I20260923 20:22:05.132961 20094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:05.172430 20094 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:05.174292 20094 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:05.187282 20094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb0bd1fad594ba486f975783049e31d: Generated new TSK 0
I20260923 20:22:05.188301 20094 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 20:22:05.227586 20110 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 20:22:05.227773 20110 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:22:05.229758 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:05.206687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.14.127.254:36055)
0923 20:22:05.206899 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:05.206906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:05.207382 (+ 476us) server_negotiation.cc:407] Connection header received
0923 20:22:05.207715 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:05.207721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:05.207808 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:05.207967 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:22:05.208523 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.209682 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.211387 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.212088 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.212219 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:05.212711 (+ 492us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:22:05.229090 (+ 16379us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 20:22:05.229457 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 20:22:05.229887 20083 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 20:22:05.231330 20109 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:05.206502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36055 (local address 127.0.0.1:38166)
0923 20:22:05.207252 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:05.207268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:05.207424 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:05.208034 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:05.208038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:05.208048 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:22:05.208367 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:05.208375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.209869 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.209878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:05.211249 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:05.211262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.212258 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.212262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:05.212381 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:05.212440 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:22:05.229372 (+ 16932us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 20:22:05.229629 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.127.254:36055: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":429,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.127.254:36055: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 20:22:05.233484 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20014
I20260923 20:22:05.234723 20075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:05.328514 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20014
2026-09-23T20:22:05Z chronyd exiting
I20260923 20:22:05.378716 14847 test_util.cc:175] -----------------------------------------------
I20260923 20:22:05.378847 14847 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194882707199-14847-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 20:22:05.382397 14847 mini_oidc.cc:152] Starting JWKS server
I20260923 20:22:05.384284 14847 webserver.cc:460] Webserver started at https://127.0.0.1:33705/ using document root <none> and password file <none>
I20260923 20:22:05.384469 14847 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:22:05.385066 14847 webserver.cc:460] Webserver started at http://127.0.0.1:39229/ using document root <none> and password file <none>
2026-09-23T20:22:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:05Z Disabled control of system clock
I20260923 20:22:05.410385 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36601
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:36601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33705/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/testchainca.cert with env {}
W20260923 20:22:05.733444 20117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:05.733711 20117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:05.733857 20117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:05.741083 20117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:05.741178 20117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:05.741251 20117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:05.741370 20117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:05.741457 20117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:22:05.741570 20117 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:22:05.741647 20117 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33705/jwks.json
I20260923 20:22:05.751689 20117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:36601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33705/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:05.752969 20117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:05.754277 20117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:05.766475 20122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:05.766428 20125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:05.766424 20123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:05.767448 20117 server_base.cc:1034] running on GCE node
I20260923 20:22:05.768213 20117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:05.769639 20117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:05.770881 20117 hybrid_clock.cc:648] HybridClock initialized: now 1790194925770839 us; error 56 us; skew 500 ppm
I20260923 20:22:05.771261 20117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:05.773609 20117 webserver.cc:460] Webserver started at http://127.14.127.254:42995/ using document root <none> and password file <none>
I20260923 20:22:05.774338 20117 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:05.774487 20117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:05.774916 20117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:05.780567 20117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "8a83efac23a046bfa8386cd11f8d7cc5"
format_stamp: "Formatted at 2026-09-23 20:22:05 on dist-test-slave-wmd0"
server_key: "5ee2d37b9b574b59c4a6ca35dfd53d93"
server_key_iv: "405e4d169ae16c9c7a6fb9c44d297732"
server_key_version: "encryptionkey@0"
I20260923 20:22:05.781656 20117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "8a83efac23a046bfa8386cd11f8d7cc5"
format_stamp: "Formatted at 2026-09-23 20:22:05 on dist-test-slave-wmd0"
server_key: "5ee2d37b9b574b59c4a6ca35dfd53d93"
server_key_iv: "405e4d169ae16c9c7a6fb9c44d297732"
server_key_version: "encryptionkey@0"
I20260923 20:22:05.787026 20117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:22:05.790496 20131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:05.791958 20117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:05.792155 20117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "8a83efac23a046bfa8386cd11f8d7cc5"
format_stamp: "Formatted at 2026-09-23 20:22:05 on dist-test-slave-wmd0"
server_key: "5ee2d37b9b574b59c4a6ca35dfd53d93"
server_key_iv: "405e4d169ae16c9c7a6fb9c44d297732"
server_key_version: "encryptionkey@0"
I20260923 20:22:05.792378 20117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:05.813731 20117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:05.814599 20117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:05.814978 20117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:05.833036 20117 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:36601
I20260923 20:22:05.833056 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:36601 every 8 connection(s)
I20260923 20:22:05.834353 20117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:05.837918 20183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:05.838680 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20117
I20260923 20:22:05.838894 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:05.839517 14847 external_mini_cluster.cc:1428] Setting key 74c8f951b17d6173ee8ce01ff5ff17b9
I20260923 20:22:05.852476 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5: Bootstrap starting.
I20260923 20:22:05.854511 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:05.841451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.14.127.254:36601)
0923 20:22:05.842543 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:05.842557 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:05.842628 (+ 71us) server_negotiation.cc:407] Connection header received
0923 20:22:05.844556 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:05.844586 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:05.845194 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:05.845658 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:05.846170 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.848090 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.849310 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.850384 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.850552 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:05.851715 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:05.851726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:05.851744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:05.852452 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:05.852554 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:05.852581 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:05.852718 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:05.852923 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 20:22:05.853249 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":341,"threads_started":1}
I20260923 20:22:05.856400 20183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:05.857667 20183 log.cc:824] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:05.858335 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:22:05.860818 20183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5: No bootstrap required, opened a new log
I20260923 20:22:05.868242 20183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a83efac23a046bfa8386cd11f8d7cc5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36601 } }
I20260923 20:22:05.868575 20183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:05.868647 20183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a83efac23a046bfa8386cd11f8d7cc5, State: Initialized, Role: FOLLOWER
I20260923 20:22:05.869174 20183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a83efac23a046bfa8386cd11f8d7cc5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36601 } }
I20260923 20:22:05.869372 20183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:05.869465 20183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:05.869589 20183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:05.871673 20183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a83efac23a046bfa8386cd11f8d7cc5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36601 } }
I20260923 20:22:05.872296 20183 leader_election.cc:302] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a83efac23a046bfa8386cd11f8d7cc5; no voters:
I20260923 20:22:05.872833 20183 leader_election.cc:288] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:05.873018 20188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:05.873713 20188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [term 1 LEADER]: Becoming Leader. State: Replica: 8a83efac23a046bfa8386cd11f8d7cc5, State: Running, Role: LEADER
I20260923 20:22:05.874221 20188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a83efac23a046bfa8386cd11f8d7cc5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36601 } }
I20260923 20:22:05.874900 20183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:05.877080 20190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a83efac23a046bfa8386cd11f8d7cc5. Latest consensus state: current_term: 1 leader_uuid: "8a83efac23a046bfa8386cd11f8d7cc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a83efac23a046bfa8386cd11f8d7cc5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36601 } } }
I20260923 20:22:05.877264 20189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a83efac23a046bfa8386cd11f8d7cc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a83efac23a046bfa8386cd11f8d7cc5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36601 } } }
I20260923 20:22:05.877511 20190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:05.877573 20189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:05.878997 20197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:05.884231 20197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:05.892831 20197 catalog_manager.cc:1347] Generated new cluster ID: dd708c6de50a426090aa00da5b80b339
I20260923 20:22:05.892951 20197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:05.910748 20197 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:05.911986 20197 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:05.921156 20197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a83efac23a046bfa8386cd11f8d7cc5: Generated new TSK 0
I20260923 20:22:05.921965 20197 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 20:22:05.954216 20213 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 20:22:05.954362 20213 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:22:05.955649 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:05.935581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.14.127.254:36601)
0923 20:22:05.935759 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:05.935765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:05.936152 (+ 387us) server_negotiation.cc:407] Connection header received
0923 20:22:05.936340 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:05.936346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:05.936458 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:05.936620 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:22:05.937036 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.938189 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.939581 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.940272 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.940390 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:05.940841 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:22:05.954949 (+ 14108us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 20:22:05.955326 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 20:22:05.955693 20212 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:05.935455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36601 (local address 127.0.0.1:44038)
0923 20:22:05.936056 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:05.936071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:05.936183 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:05.936632 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:05.936635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:05.936644 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:22:05.936859 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:05.936866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.938329 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.938333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:05.939446 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:05.939457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:05.940425 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:05.940430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:05.940565 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:05.940596 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:22:05.955182 (+ 14586us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 20:22:05.955385 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.127.254:36601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":329,"threads_started":1}
W20260923 20:22:05.955783 20186 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44038: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260923 20:22:05.956385 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20117
I20260923 20:22:05.957600 20178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:06.054441 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20117
2026-09-23T20:22:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (728 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 20:22:06.111088 14847 mini_oidc.cc:152] Starting JWKS server
I20260923 20:22:06.112887 14847 webserver.cc:460] Webserver started at https://127.0.0.1:36679/ using document root <none> and password file <none>
I20260923 20:22:06.113137 14847 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:22:06.113744 14847 webserver.cc:460] Webserver started at http://127.0.0.1:44627/ using document root <none> and password file <none>
2026-09-23T20:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:06Z Disabled control of system clock
I20260923 20:22:06.139134 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:38245
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:37693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:38245
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36679/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 20:22:06.452560 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:06.452839 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:06.452922 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:06.459996 20220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:06.460098 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:06.460135 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:06.460186 20220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:06.460230 20220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:22:06.460283 20220 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:22:06.460322 20220 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36679/jwks.json
I20260923 20:22:06.469187 20220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:38245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36679/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:06.470398 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:06.472077 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:06.483991 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:06.483943 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:06.484354 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:06.484707 20220 server_base.cc:1034] running on GCE node
I20260923 20:22:06.485494 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:06.486925 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:06.488168 20220 hybrid_clock.cc:648] HybridClock initialized: now 1790194926488135 us; error 39 us; skew 500 ppm
I20260923 20:22:06.488623 20220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:06.490937 20220 webserver.cc:460] Webserver started at http://127.14.127.254:36125/ using document root <none> and password file <none>
I20260923 20:22:06.491631 20220 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:06.491766 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:06.492102 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:06.494946 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "8216c8e0654c4b69886c005bc44b2e5c"
format_stamp: "Formatted at 2026-09-23 20:22:06 on dist-test-slave-wmd0"
server_key: "13e470e4023aa99f74bfa7b3e0427a23"
server_key_iv: "fabfb85f48c16e5f2232ce0e50215e99"
server_key_version: "encryptionkey@0"
I20260923 20:22:06.495749 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "8216c8e0654c4b69886c005bc44b2e5c"
format_stamp: "Formatted at 2026-09-23 20:22:06 on dist-test-slave-wmd0"
server_key: "13e470e4023aa99f74bfa7b3e0427a23"
server_key_iv: "fabfb85f48c16e5f2232ce0e50215e99"
server_key_version: "encryptionkey@0"
I20260923 20:22:06.500792 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:22:06.504416 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:06.505838 20220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:22:06.506048 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "8216c8e0654c4b69886c005bc44b2e5c"
format_stamp: "Formatted at 2026-09-23 20:22:06 on dist-test-slave-wmd0"
server_key: "13e470e4023aa99f74bfa7b3e0427a23"
server_key_iv: "fabfb85f48c16e5f2232ce0e50215e99"
server_key_version: "encryptionkey@0"
I20260923 20:22:06.506278 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:06.523204 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:06.523998 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:06.524312 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:06.542143 20220 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:38245
I20260923 20:22:06.542168 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:38245 every 8 connection(s)
I20260923 20:22:06.543560 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:06.546078 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20220
I20260923 20:22:06.546299 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:06.547102 14847 external_mini_cluster.cc:1428] Setting key 39ce5ace281083b55e958d99ca685009
I20260923 20:22:06.548089 20286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:06.562858 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:06.548971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.14.127.254:38245)
0923 20:22:06.550281 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:06.550301 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:06.551220 (+ 919us) server_negotiation.cc:407] Connection header received
0923 20:22:06.552897 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:06.552929 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:06.553517 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:06.553969 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:06.554522 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:06.556497 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:06.557838 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:06.558901 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:06.559096 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:06.560150 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:06.560161 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:06.560183 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:06.560901 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:06.560995 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:06.561025 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:06.561130 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:06.561291 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 20:22:06.561616 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":385,"threads_started":1}
I20260923 20:22:06.562857 20286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c: Bootstrap starting.
I20260923 20:22:06.566435 20286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:06.566602 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:22:06.567519 20286 log.cc:824] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:06.570396 20286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c: No bootstrap required, opened a new log
I20260923 20:22:06.577445 20286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216c8e0654c4b69886c005bc44b2e5c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38245 } }
I20260923 20:22:06.577910 20286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:06.578034 20286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8216c8e0654c4b69886c005bc44b2e5c, State: Initialized, Role: FOLLOWER
I20260923 20:22:06.578569 20286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216c8e0654c4b69886c005bc44b2e5c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38245 } }
I20260923 20:22:06.578929 20286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:06.579079 20286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:06.579268 20286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:06.581480 20286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216c8e0654c4b69886c005bc44b2e5c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38245 } }
I20260923 20:22:06.581915 20286 leader_election.cc:302] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8216c8e0654c4b69886c005bc44b2e5c; no voters:
I20260923 20:22:06.582432 20286 leader_election.cc:288] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:06.582638 20291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:06.583638 20291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [term 1 LEADER]: Becoming Leader. State: Replica: 8216c8e0654c4b69886c005bc44b2e5c, State: Running, Role: LEADER
I20260923 20:22:06.584199 20291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216c8e0654c4b69886c005bc44b2e5c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38245 } }
I20260923 20:22:06.584731 20286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:06.586915 20292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8216c8e0654c4b69886c005bc44b2e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216c8e0654c4b69886c005bc44b2e5c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38245 } } }
I20260923 20:22:06.587280 20292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:06.587833 20293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8216c8e0654c4b69886c005bc44b2e5c. Latest consensus state: current_term: 1 leader_uuid: "8216c8e0654c4b69886c005bc44b2e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8216c8e0654c4b69886c005bc44b2e5c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38245 } } }
I20260923 20:22:06.588136 20293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:06.592168 20300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:06.596477 20300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:06.604763 20300 catalog_manager.cc:1347] Generated new cluster ID: 6a9184e2f7f6463b9ed6dbc999a27066
I20260923 20:22:06.604884 20300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:06.612118 20300 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:06.613281 20300 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:06.619130 20300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8216c8e0654c4b69886c005bc44b2e5c: Generated new TSK 0
I20260923 20:22:06.620124 20300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:22:06.650487 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:06.642080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.14.127.254:38245)
0923 20:22:06.642288 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:06.642295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:06.642734 (+ 439us) server_negotiation.cc:407] Connection header received
0923 20:22:06.642955 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:06.642961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:06.643052 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:06.643231 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:22:06.643648 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:06.644769 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:06.646163 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:06.646882 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:06.647019 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:06.647515 (+ 496us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:22:06.649856 (+ 2341us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36679/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 20:22:06.650170 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40034: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36679/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:22:06.650641 20289 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40034: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36679/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 20:22:06.652386 20316 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:06.641921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:38245 (local address 127.0.0.1:40034)
0923 20:22:06.642589 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:06.642647 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:06.642773 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:06.643237 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:06.643240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:06.643249 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:22:06.643503 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:06.643510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:06.644903 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:06.644907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:06.646028 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:06.646039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:06.647067 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:06.647072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:06.647198 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:06.647229 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:22:06.650063 (+ 2834us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36679/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 20:22:06.651952 (+ 1889us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.127.254:38245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36679/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":1573,"thread_start_us":387,"threads_started":1}
I20260923 20:22:06.653685 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20220
I20260923 20:22:06.654928 20281 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 20:22:06.664609 20317 webserver.cc:533] Webserver: sslize failed:
E20260923 20:22:06.664726 20317 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:22:06.749951 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20220
2026-09-23T20:22:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (693 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T20:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:06Z Disabled control of system clock
I20260923 20:22:06.830475 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37343
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:37343
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/testchainca.cert with env {}
W20260923 20:22:07.145938 20322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:07.146222 20322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:07.146344 20322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:07.153551 20322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:07.153651 20322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:07.153688 20322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:07.153738 20322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:07.153782 20322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:22:07.153836 20322 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:22:07.153873 20322 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 20:22:07.163244 20322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:37343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:07.164410 20322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:07.165592 20322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:07.177023 20330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:07.177091 20328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:07.177062 20327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:07.177874 20322 server_base.cc:1034] running on GCE node
I20260923 20:22:07.178613 20322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:07.180020 20322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:07.181244 20322 hybrid_clock.cc:648] HybridClock initialized: now 1790194927181175 us; error 67 us; skew 500 ppm
I20260923 20:22:07.181571 20322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:07.183821 20322 webserver.cc:460] Webserver started at http://127.14.127.254:37315/ using document root <none> and password file <none>
I20260923 20:22:07.184492 20322 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:07.184630 20322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:07.184942 20322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:07.187768 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "221a62c4a7b240488a5b80afca4e18f2"
format_stamp: "Formatted at 2026-09-23 20:22:07 on dist-test-slave-wmd0"
server_key: "43c464eb95fbf9462d1cd928aa99f5b8"
server_key_iv: "a81eae1b5be5d1e5b770473611d57990"
server_key_version: "encryptionkey@0"
I20260923 20:22:07.188560 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "221a62c4a7b240488a5b80afca4e18f2"
format_stamp: "Formatted at 2026-09-23 20:22:07 on dist-test-slave-wmd0"
server_key: "43c464eb95fbf9462d1cd928aa99f5b8"
server_key_iv: "a81eae1b5be5d1e5b770473611d57990"
server_key_version: "encryptionkey@0"
I20260923 20:22:07.193470 20322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:22:07.196702 20336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:07.197999 20322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:22:07.198175 20322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "221a62c4a7b240488a5b80afca4e18f2"
format_stamp: "Formatted at 2026-09-23 20:22:07 on dist-test-slave-wmd0"
server_key: "43c464eb95fbf9462d1cd928aa99f5b8"
server_key_iv: "a81eae1b5be5d1e5b770473611d57990"
server_key_version: "encryptionkey@0"
I20260923 20:22:07.198383 20322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:07.220769 20322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:07.221601 20322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:07.221901 20322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:07.239833 20322 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:37343
I20260923 20:22:07.239837 20387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:37343 every 8 connection(s)
I20260923 20:22:07.241178 20322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:07.244719 20388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:07.248329 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20322
I20260923 20:22:07.248528 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:07.249146 14847 external_mini_cluster.cc:1428] Setting key 69ee4ec1bfd1d36c0736f30280b3df92
I20260923 20:22:07.257117 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2: Bootstrap starting.
I20260923 20:22:07.260679 20388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:07.261976 20388 log.cc:824] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:07.265184 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:07.251075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.14.127.254:37343)
0923 20:22:07.252209 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:07.252226 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:07.252279 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:22:07.255228 (+ 2949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:07.255257 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:07.255849 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:07.256276 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:07.256818 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:07.258837 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:07.260099 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:07.261206 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:07.261398 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:07.262584 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:07.262597 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:07.262644 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:07.263184 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:07.263306 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:07.263337 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:07.263452 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:07.263678 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 20:22:07.263978 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":346,"threads_started":1}
I20260923 20:22:07.265302 20388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2: No bootstrap required, opened a new log
I20260923 20:22:07.268961 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:22:07.273381 20388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221a62c4a7b240488a5b80afca4e18f2" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37343 } }
I20260923 20:22:07.273850 20388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:07.273968 20388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221a62c4a7b240488a5b80afca4e18f2, State: Initialized, Role: FOLLOWER
I20260923 20:22:07.274507 20388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221a62c4a7b240488a5b80afca4e18f2" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37343 } }
I20260923 20:22:07.274816 20388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:07.274966 20388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:07.275139 20388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:07.277225 20388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221a62c4a7b240488a5b80afca4e18f2" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37343 } }
I20260923 20:22:07.277789 20388 leader_election.cc:302] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221a62c4a7b240488a5b80afca4e18f2; no voters:
I20260923 20:22:07.278399 20388 leader_election.cc:288] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:07.278550 20393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:07.279284 20393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [term 1 LEADER]: Becoming Leader. State: Replica: 221a62c4a7b240488a5b80afca4e18f2, State: Running, Role: LEADER
I20260923 20:22:07.279845 20393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221a62c4a7b240488a5b80afca4e18f2" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37343 } }
I20260923 20:22:07.280441 20388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:07.282730 20394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221a62c4a7b240488a5b80afca4e18f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221a62c4a7b240488a5b80afca4e18f2" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37343 } } }
I20260923 20:22:07.283136 20394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:07.283907 20395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221a62c4a7b240488a5b80afca4e18f2. Latest consensus state: current_term: 1 leader_uuid: "221a62c4a7b240488a5b80afca4e18f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221a62c4a7b240488a5b80afca4e18f2" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37343 } } }
I20260923 20:22:07.284292 20395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:07.288072 20400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:07.292611 20400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:07.300437 20400 catalog_manager.cc:1347] Generated new cluster ID: 5768d3fc98d846a29aaf7988d29e195c
I20260923 20:22:07.300550 20400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:07.316856 20400 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:07.318113 20400 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:07.325525 20400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221a62c4a7b240488a5b80afca4e18f2: Generated new TSK 0
I20260923 20:22:07.326273 20400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:22:07.388852 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:07.343240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.14.127.254:37343)
0923 20:22:07.343422 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:07.343428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:07.343747 (+ 319us) server_negotiation.cc:407] Connection header received
0923 20:22:07.343930 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:07.343936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:07.344042 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:07.344214 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:22:07.344622 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:07.345756 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:07.347341 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:07.348051 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:07.348175 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:07.348592 (+ 417us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:22:07.388094 (+ 39502us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 20:22:07.388511 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36744: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 20:22:07.389000 20391 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36744: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 20:22:07.389178 20417 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:07.343069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:37343 (local address 127.0.0.1:36744)
0923 20:22:07.343645 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:07.343659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:07.343774 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:07.344217 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:07.344221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:07.344230 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:22:07.344476 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:07.344483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:07.345895 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:07.345899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:07.347188 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:07.347201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:07.348207 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:07.348211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:07.348324 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:07.348355 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:22:07.388393 (+ 40038us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 20:22:07.388639 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.127.254:37343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":291,"threads_started":1}
I20260923 20:22:07.390403 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20322
I20260923 20:22:07.391587 20383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:07.487835 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20322
2026-09-23T20:22:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (714 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (401 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (413 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:22:08 dist-test-slave-wmd0 krb5kdc[20432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:22:08 dist-test-slave-wmd0 krb5kdc[20432](info): set up 2 sockets
Sep 23 20:22:08 dist-test-slave-wmd0 krb5kdc[20432](info): commencing operation
krb5kdc: starting...
W20260923 20:22:09.415426 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:22:09 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:22:09.620548 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:34903
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:34903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:09.942673 20445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:09.943073 20445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:09.943238 20445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:09.950758 20445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:09.950937 20445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:09.951047 20445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:09.951156 20445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:09.951231 20445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:09.961310 20445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:34903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:34903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:09.962767 20445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:09.964109 20445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:09.976195 20453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:09.976151 20450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:09.976235 20451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:09.977366 20445 server_base.cc:1034] running on GCE node
I20260923 20:22:09.978204 20445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:09.979729 20445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:09.980993 20445 hybrid_clock.cc:648] HybridClock initialized: now 1790194929980947 us; error 58 us; skew 500 ppm
I20260923 20:22:09.981375 20445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:09 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194929, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:09.984508 20445 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:22:09.986043 20445 webserver.cc:460] Webserver started at http://127.14.127.254:45255/ using document root <none> and password file <none>
I20260923 20:22:09.986829 20445 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:09.986990 20445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:09.987327 20445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:09.990324 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "7c88d438f952479097e2dc7b0430ab5e"
format_stamp: "Formatted at 2026-09-23 20:22:09 on dist-test-slave-wmd0"
server_key: "31d58c2aee6b2eddc3065328331fc47e"
server_key_iv: "5af67165870e2078b06a02741bf5e4ed"
server_key_version: "encryptionkey@0"
I20260923 20:22:09.991183 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "7c88d438f952479097e2dc7b0430ab5e"
format_stamp: "Formatted at 2026-09-23 20:22:09 on dist-test-slave-wmd0"
server_key: "31d58c2aee6b2eddc3065328331fc47e"
server_key_iv: "5af67165870e2078b06a02741bf5e4ed"
server_key_version: "encryptionkey@0"
I20260923 20:22:09.996495 20445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:22:10.000025 20460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:10.001513 20445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:10.001715 20445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "7c88d438f952479097e2dc7b0430ab5e"
format_stamp: "Formatted at 2026-09-23 20:22:09 on dist-test-slave-wmd0"
server_key: "31d58c2aee6b2eddc3065328331fc47e"
server_key_iv: "5af67165870e2078b06a02741bf5e4ed"
server_key_version: "encryptionkey@0"
I20260923 20:22:10.001925 20445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:10.031343 20445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:10.035519 20445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:10.035820 20445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:10.054507 20445 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:34903
I20260923 20:22:10.054591 20511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:34903 every 8 connection(s)
I20260923 20:22:10.055962 20445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:10.059798 20512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:10.060400 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20445
I20260923 20:22:10.060621 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:10.061286 14847 external_mini_cluster.cc:1428] Setting key 1bffa600c44104f7e92c79021935ee54
I20260923 20:22:10.074186 20512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e: Bootstrap starting.
Sep 23 20:22:10 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:10.077669 20512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:10.079133 20512 log.cc:824] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:10.082472 20512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e: No bootstrap required, opened a new log
I20260923 20:22:10.085927 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:10.063410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.14.127.254:34903)
0923 20:22:10.064191 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:10.064208 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:10.064271 (+ 63us) server_negotiation.cc:407] Connection header received
0923 20:22:10.066444 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:10.066475 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:10.067024 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:10.067496 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:10.068821 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:10.070838 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:10.072045 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:10.073085 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:10.073259 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:10.077434 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:10.077449 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:10.077479 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:10.077541 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:10.081334 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:10.082041 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:10.082049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:10.082057 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:10.082259 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:10.082661 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:10.082665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:10.082668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:10.083409 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:10.083686 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:10.084060 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 20:22:10.084526 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":325,"threads_started":1}
I20260923 20:22:10.091753 20512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c88d438f952479097e2dc7b0430ab5e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34903 } }
I20260923 20:22:10.092094 20512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:10.092170 20512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c88d438f952479097e2dc7b0430ab5e, State: Initialized, Role: FOLLOWER
I20260923 20:22:10.092639 20512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c88d438f952479097e2dc7b0430ab5e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34903 } }
I20260923 20:22:10.092832 20512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:10.092933 20512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:10.093052 20512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:10.095583 20512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c88d438f952479097e2dc7b0430ab5e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34903 } }
I20260923 20:22:10.096078 20512 leader_election.cc:302] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c88d438f952479097e2dc7b0430ab5e; no voters:
I20260923 20:22:10.096585 20512 leader_election.cc:288] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:10.096730 20517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:10.097426 20517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [term 1 LEADER]: Becoming Leader. State: Replica: 7c88d438f952479097e2dc7b0430ab5e, State: Running, Role: LEADER
I20260923 20:22:10.098032 20517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c88d438f952479097e2dc7b0430ab5e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34903 } }
I20260923 20:22:10.098727 20512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:10.101109 20519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c88d438f952479097e2dc7b0430ab5e. Latest consensus state: current_term: 1 leader_uuid: "7c88d438f952479097e2dc7b0430ab5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c88d438f952479097e2dc7b0430ab5e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34903 } } }
I20260923 20:22:10.101475 20519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:10.102205 20518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c88d438f952479097e2dc7b0430ab5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c88d438f952479097e2dc7b0430ab5e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34903 } } }
I20260923 20:22:10.102665 20518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:10.103557 20525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:10.108623 20525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:10.117478 20525 catalog_manager.cc:1347] Generated new cluster ID: b4c7aeb85aed4d2bb878496ef9e5f09b
I20260923 20:22:10.117605 20525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:10.136785 20525 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:10.138334 20525 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:10.151775 20525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c88d438f952479097e2dc7b0430ab5e: Generated new TSK 0
I20260923 20:22:10.152781 20525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:22:10.211203 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:34903
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:10.532905 20538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:10.533179 20538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:10.533272 20538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:10.540475 20538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:10.540589 20538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:10.540699 20538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:10.550077 20538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:34903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:10.551319 20538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:10.552546 20538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:10.565397 20546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:10.565290 20543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:10.565320 20544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:10.566061 20538 server_base.cc:1034] running on GCE node
I20260923 20:22:10.566702 20538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:10.567643 20538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:10.568866 20538 hybrid_clock.cc:648] HybridClock initialized: now 1790194930568827 us; error 48 us; skew 500 ppm
I20260923 20:22:10.569155 20538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:10 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194930, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:10.572155 20538 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:22:10.573690 20538 webserver.cc:460] Webserver started at http://127.14.127.193:34143/ using document root <none> and password file <none>
I20260923 20:22:10.574352 20538 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:10.574487 20538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:10.574833 20538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:10.577610 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "c32b73f88fbd488094909cdf812bf408"
format_stamp: "Formatted at 2026-09-23 20:22:10 on dist-test-slave-wmd0"
server_key: "91be680f47561c9ff2eeef316b903be3"
server_key_iv: "22c25d3847e7a0e9cd65413923f61eca"
server_key_version: "encryptionkey@0"
I20260923 20:22:10.578430 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "c32b73f88fbd488094909cdf812bf408"
format_stamp: "Formatted at 2026-09-23 20:22:10 on dist-test-slave-wmd0"
server_key: "91be680f47561c9ff2eeef316b903be3"
server_key_iv: "22c25d3847e7a0e9cd65413923f61eca"
server_key_version: "encryptionkey@0"
I20260923 20:22:10.583258 20538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:22:10.586522 20553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:10.587831 20538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:22:10.588001 20538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "c32b73f88fbd488094909cdf812bf408"
format_stamp: "Formatted at 2026-09-23 20:22:10 on dist-test-slave-wmd0"
server_key: "91be680f47561c9ff2eeef316b903be3"
server_key_iv: "22c25d3847e7a0e9cd65413923f61eca"
server_key_version: "encryptionkey@0"
I20260923 20:22:10.588192 20538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:10.607115 20538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:10.611547 20538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:10.611840 20538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:10.612749 20538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:10.614049 20538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:10.614154 20538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:10.614306 20538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:10.614379 20538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:10.646795 20538 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:34339
I20260923 20:22:10.646813 20665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:34339 every 8 connection(s)
I20260923 20:22:10.648113 20538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:10.649780 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20538
I20260923 20:22:10.649948 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:10.650545 14847 external_mini_cluster.cc:1428] Setting key bb9442256d7c36b5d8c4c51b41ba11c9
Sep 23 20:22:10 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194930, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:22:10.672667 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:10.652136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:58167 (local address 127.14.127.254:34903)
0923 20:22:10.652586 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:10.652593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:10.653785 (+ 1192us) server_negotiation.cc:407] Connection header received
0923 20:22:10.656665 (+ 2880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:10.656669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:10.656734 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:10.656830 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:10.658902 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:10.659593 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:10.661122 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:10.661925 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:10.662067 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:10.665607 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:10.665616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:10.665624 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:10.665669 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:10.669357 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:10.670170 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:10.670177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:10.670180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:10.670333 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:10.670764 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:10.670772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:10.670776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:10.671167 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:10.671331 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:10.671965 (+ 634us) server_negotiation.cc:299] Negotiation successful
0923 20:22:10.672214 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":217,"threads_started":1}
I20260923 20:22:10.673331 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:10.652711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:34903 (local address 127.14.127.193:58167)
0923 20:22:10.653603 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:10.653660 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:10.656381 (+ 2721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:10.657116 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:10.657130 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:10.657897 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:10.658735 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:10.658762 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:10.659774 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:10.659777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:10.660980 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:10.660991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:10.662123 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:10.662127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:10.662312 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:10.663014 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:10.663035 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:10.665386 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:10.669590 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:10.669596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:10.669608 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:10.670026 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:10.670437 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:10.670440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:10.670443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:10.670603 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:10.671283 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:10.671288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:10.671561 (+ 273us) client_negotiation.cc:769] Sending connection context
0923 20:22:10.671785 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 20:22:10.672179 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":348,"threads_started":1}
I20260923 20:22:10.674918 20666 heartbeater.cc:347] Connected to a master server at 127.14.127.254:34903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
I20260923 20:22:10.675242 20666 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:10.676103 20666 heartbeater.cc:511] Master 127.14.127.254:34903 requested a full tablet report, sending...
I20260923 20:22:10.678498 20477 ts_manager.cc:194] Registered new tserver with Master: c32b73f88fbd488094909cdf812bf408 (127.14.127.193:34339)
I20260923 20:22:10.680670 20477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:58167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:22:10.699815 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:34903
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:11.015877 20672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:11.016152 20672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:11.016309 20672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:11.023386 20672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:11.023504 20672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:11.023679 20672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:11.033200 20672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:34903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:11.034483 20672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:11.035835 20672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:11.048600 20677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:11.048640 20678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:11.048877 20680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:11.049552 20672 server_base.cc:1034] running on GCE node
I20260923 20:22:11.050122 20672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:11.051083 20672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:11.052314 20672 hybrid_clock.cc:648] HybridClock initialized: now 1790194931052262 us; error 47 us; skew 500 ppm
I20260923 20:22:11.052628 20672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.055912 20672 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:22:11.057456 20672 webserver.cc:460] Webserver started at http://127.14.127.194:35689/ using document root <none> and password file <none>
I20260923 20:22:11.058112 20672 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:11.058254 20672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:11.058552 20672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:11.061301 20672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "b57bfc9890054277a93456650b5d0acd"
format_stamp: "Formatted at 2026-09-23 20:22:11 on dist-test-slave-wmd0"
server_key: "826c817263c5d7d6f3d067e15dd14830"
server_key_iv: "1ecd2e472db86f2c49e32514328e1567"
server_key_version: "encryptionkey@0"
I20260923 20:22:11.062070 20672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "b57bfc9890054277a93456650b5d0acd"
format_stamp: "Formatted at 2026-09-23 20:22:11 on dist-test-slave-wmd0"
server_key: "826c817263c5d7d6f3d067e15dd14830"
server_key_iv: "1ecd2e472db86f2c49e32514328e1567"
server_key_version: "encryptionkey@0"
I20260923 20:22:11.067130 20672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:22:11.070590 20687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:11.071835 20672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:11.072016 20672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "b57bfc9890054277a93456650b5d0acd"
format_stamp: "Formatted at 2026-09-23 20:22:11 on dist-test-slave-wmd0"
server_key: "826c817263c5d7d6f3d067e15dd14830"
server_key_iv: "1ecd2e472db86f2c49e32514328e1567"
server_key_version: "encryptionkey@0"
I20260923 20:22:11.072219 20672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:11.089982 20672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:11.093940 20672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:11.094209 20672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:11.095165 20672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:11.096401 20672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:11.096504 20672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:11.096665 20672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:11.096735 20672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:11.128083 20672 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:44583
I20260923 20:22:11.128124 20799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:44583 every 8 connection(s)
I20260923 20:22:11.129343 20672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:11.137295 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20672
I20260923 20:22:11.137466 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:11.137907 14847 external_mini_cluster.cc:1428] Setting key a846ab5849effdfcd9fa4dcb77fb621a
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:11.150051 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.131957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:58471 (local address 127.14.127.254:34903)
0923 20:22:11.132142 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.132147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.133316 (+ 1169us) server_negotiation.cc:407] Connection header received
0923 20:22:11.135079 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.135084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.135149 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.135254 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.136654 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.137409 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.139169 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.139703 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.139867 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.143444 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:11.143449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:11.143453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:11.143486 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:11.146769 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.147471 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.147479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.147486 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.147648 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.148037 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.148044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.148050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.148424 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:11.148585 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:11.149199 (+ 614us) server_negotiation.cc:299] Negotiation successful
0923 20:22:11.149455 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:22:11.150461 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.132473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:34903 (local address 127.14.127.194:58471)
0923 20:22:11.133156 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:11.133199 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:11.134828 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:11.135481 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:11.135488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:11.135986 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:11.136507 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:11.136523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.137574 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.137579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:11.139025 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:11.139037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.139955 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.139964 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:11.140183 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.140852 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:11.140877 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:11.143206 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:11.146911 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:11.146918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:11.146931 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:11.147339 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:11.147752 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:11.147755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:11.147758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:11.147924 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:11.148541 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:11.148546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:11.148837 (+ 291us) client_negotiation.cc:769] Sending connection context
0923 20:22:11.149050 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 20:22:11.149400 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":277,"threads_started":1}
I20260923 20:22:11.151834 20800 heartbeater.cc:347] Connected to a master server at 127.14.127.254:34903
I20260923 20:22:11.152146 20800 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:11.152832 20800 heartbeater.cc:511] Master 127.14.127.254:34903 requested a full tablet report, sending...
I20260923 20:22:11.154302 20477 ts_manager.cc:194] Registered new tserver with Master: b57bfc9890054277a93456650b5d0acd (127.14.127.194:44583)
I20260923 20:22:11.155637 20477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:58471
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:22:11.185564 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:34903
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:40621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:11.499704 20805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:11.499980 20805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:11.500072 20805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:11.507822 20805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:11.507941 20805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:11.508054 20805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:11.517601 20805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:40621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:34903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:11.518816 20805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:11.520040 20805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:11.533077 20810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:11.532997 20811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:11.533028 20813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:11.534096 20805 server_base.cc:1034] running on GCE node
I20260923 20:22:11.534724 20805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:11.535550 20805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:11.536777 20805 hybrid_clock.cc:648] HybridClock initialized: now 1790194931536731 us; error 55 us; skew 500 ppm
I20260923 20:22:11.537074 20805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.540095 20805 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:22:11.541464 20805 webserver.cc:460] Webserver started at http://127.14.127.195:36859/ using document root <none> and password file <none>
I20260923 20:22:11.542106 20805 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:11.542248 20805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:11.542552 20805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:11.545311 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "aff421d7599349d7bf65a9f82e5f7781"
format_stamp: "Formatted at 2026-09-23 20:22:11 on dist-test-slave-wmd0"
server_key: "5a150f5f0cffbf66fc0937b4ed726d77"
server_key_iv: "80b49096eec755f87ba4dbad76e7a8bd"
server_key_version: "encryptionkey@0"
I20260923 20:22:11.546087 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "aff421d7599349d7bf65a9f82e5f7781"
format_stamp: "Formatted at 2026-09-23 20:22:11 on dist-test-slave-wmd0"
server_key: "5a150f5f0cffbf66fc0937b4ed726d77"
server_key_iv: "80b49096eec755f87ba4dbad76e7a8bd"
server_key_version: "encryptionkey@0"
I20260923 20:22:11.550994 20805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:22:11.554163 20820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:11.555411 20805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:11.555595 20805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "aff421d7599349d7bf65a9f82e5f7781"
format_stamp: "Formatted at 2026-09-23 20:22:11 on dist-test-slave-wmd0"
server_key: "5a150f5f0cffbf66fc0937b4ed726d77"
server_key_iv: "80b49096eec755f87ba4dbad76e7a8bd"
server_key_version: "encryptionkey@0"
I20260923 20:22:11.555804 20805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:11.572897 20805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:11.576996 20805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:11.577260 20805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:11.578146 20805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:11.579465 20805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:11.579566 20805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:11.579715 20805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:11.579787 20805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:11.612794 20805 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:35807
I20260923 20:22:11.612850 20932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:35807 every 8 connection(s)
I20260923 20:22:11.614073 20805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:11.623749 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20805
I20260923 20:22:11.623930 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:11.624402 14847 external_mini_cluster.cc:1428] Setting key 703f257526d5954cd6231d9ec758475d
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:11.633688 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.616753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:39589 (local address 127.14.127.254:34903)
0923 20:22:11.616932 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.616938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.618146 (+ 1208us) server_negotiation.cc:407] Connection header received
0923 20:22:11.619999 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.620005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.620099 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.620209 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.621663 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.622407 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.624010 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.624445 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.624579 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.628287 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:11.628292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:11.628295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:11.628350 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:11.631087 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.631737 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.631740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.631743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.631865 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.632229 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.632232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.632234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.632544 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:11.632654 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:11.633242 (+ 588us) server_negotiation.cc:299] Negotiation successful
0923 20:22:11.633432 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:22:11.634562 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.617256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:34903 (local address 127.14.127.195:39589)
0923 20:22:11.617985 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:11.618022 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:11.619724 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:11.620447 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:11.620455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:11.620954 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:11.621492 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:11.621509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.622576 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.622580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:11.623854 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:11.623870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.624821 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.624825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:11.625014 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.625822 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:11.625848 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:11.628104 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:11.631215 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:11.631220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:11.631231 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:11.631628 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:11.631956 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:11.631959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:11.631961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:11.632116 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:11.632647 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:11.632652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:11.632947 (+ 295us) client_negotiation.cc:769] Sending connection context
0923 20:22:11.633156 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 20:22:11.633495 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":307,"threads_started":1}
I20260923 20:22:11.635972 20933 heartbeater.cc:347] Connected to a master server at 127.14.127.254:34903
I20260923 20:22:11.636269 20933 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:11.636998 20933 heartbeater.cc:511] Master 127.14.127.254:34903 requested a full tablet report, sending...
I20260923 20:22:11.638491 20477 ts_manager.cc:194] Registered new tserver with Master: aff421d7599349d7bf65a9f82e5f7781 (127.14.127.195:35807)
I20260923 20:22:11.639696 20477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:39589
I20260923 20:22:11.639966 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/test-admin.keytab.
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.668350 14847 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 20:22:11.673321 20940 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:11.670919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.14.127.193:34339)
0923 20:22:11.671549 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.671556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.671580 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:22:11.671641 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.671647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.671834 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.672005 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.672892 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38498: BlockingRecv error: recv got EOF from 127.0.0.1:38498 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":308,"threads_started":1}
W20260923 20:22:11.673417 14847 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.675052 14847 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.676541 14847 init.cc:295] Successfully reacquired a new kerberos TGT
I20260923 20:22:11.683559 20666 heartbeater.cc:503] Master 127.14.127.254:34903 was elected leader, sending a full tablet report...
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:22:11.691527 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.678030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.14.127.193:34339)
0923 20:22:11.678190 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.678195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.678327 (+ 132us) server_negotiation.cc:407] Connection header received
0923 20:22:11.678503 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.678506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.678556 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.678690 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.679598 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.680521 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.681663 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.682210 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.682300 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.686254 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:11.686264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:11.686277 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:11.686305 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:11.688962 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.689523 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.689529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.689531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.689635 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.690020 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.690023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.690025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.690326 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:11.690580 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:11.690926 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 20:22:11.691142 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 20:22:11.697044 20940 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:11.695347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.14.127.193:34339)
0923 20:22:11.695548 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.695554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.695770 (+ 216us) server_negotiation.cc:407] Connection header received
0923 20:22:11.695958 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.695961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.696010 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.696105 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.696814 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38504: BlockingRecv error: recv got EOF from 127.0.0.1:38504 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 20:22:11.697369 14847 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.698741 14847 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.700059 14847 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:22:11.713152 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.701194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.14.127.193:34339)
0923 20:22:11.701417 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.701423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.701575 (+ 152us) server_negotiation.cc:407] Connection header received
0923 20:22:11.701676 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.701679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.701724 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.701796 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.702668 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.703407 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.704625 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.705033 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.705121 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.708566 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:11.708570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:11.708573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:11.708600 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:11.711156 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.711634 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.711637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.711639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.711744 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.712089 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.712091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.712093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.712313 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:11.712500 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:11.712695 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 20:22:11.712858 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 20:22:11.717002 20940 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:11.715563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.14.127.193:34339)
0923 20:22:11.715763 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.715768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.715918 (+ 150us) server_negotiation.cc:407] Connection header received
0923 20:22:11.716029 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.716033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.716091 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.716163 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.716751 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38522: BlockingRecv error: recv got EOF from 127.0.0.1:38522 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 20:22:11.717051 14847 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194882707199-14847-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.718374 14847 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:11.719599 14847 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:22:11.731925 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.720736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.14.127.193:34339)
0923 20:22:11.720949 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.720954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.721205 (+ 251us) server_negotiation.cc:407] Connection header received
0923 20:22:11.721293 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.721297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.721342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.721418 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.722244 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.722993 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.724088 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.724481 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.724567 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.727665 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:11.727669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:11.727672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:11.727698 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:11.729891 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.730403 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.730406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.730408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.730510 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.730861 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.730863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.730865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.731118 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:11.731322 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:11.731463 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 20:22:11.731628 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:22:11.733283 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20538
I20260923 20:22:11.744731 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.733860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.14.127.193:34339)
0923 20:22:11.734008 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.734012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.734387 (+ 375us) server_negotiation.cc:407] Connection header received
0923 20:22:11.734498 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.734501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.734548 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.734675 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.735557 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.736197 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.737276 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.737684 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.737767 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.739557 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:11.739561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:11.739564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:11.739589 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:11.742115 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.742757 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.742762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.742766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.742975 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.743449 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.743452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.743454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.743690 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:11.743946 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:11.744241 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 20:22:11.744444 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:22:11.745247 20661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:11.827505 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20538
I20260923 20:22:11.843550 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20672
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:22:11.859079 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.844542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.14.127.194:44583)
0923 20:22:11.845053 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.845059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.845084 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:22:11.845148 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.845155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.845352 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.845492 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.846906 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.847974 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.849150 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.849698 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.849809 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.853343 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:11.853353 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:11.853372 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:11.853402 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:11.856295 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.856858 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.856865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.856867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.856989 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.857371 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.857376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.857379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.857712 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:11.857938 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:11.858238 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 20:22:11.858497 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260923 20:22:11.860369 20795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:11.941797 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20672
I20260923 20:22:11.957597 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20805
Sep 23 20:22:11 dist-test-slave-wmd0 krb5kdc[20432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:22:11.973688 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:11.958815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.14.127.195:35807)
0923 20:22:11.959484 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:11.959491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:11.959549 (+ 58us) server_negotiation.cc:407] Connection header received
0923 20:22:11.959689 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:11.959697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:11.959947 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:11.960093 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:11.961216 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.962293 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.963610 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:11.964173 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:11.964272 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:11.968099 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:11.968110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:11.968125 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:11.968159 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:11.971068 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.971707 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.971714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.971716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.971840 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:11.972199 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:11.972203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:11.972205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:11.972557 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:11.972780 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:11.973105 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 20:22:11.973378 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":341,"threads_started":1}
I20260923 20:22:11.975155 20928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:12.063722 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20805
I20260923 20:22:12.079499 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20445
I20260923 20:22:12.080740 20507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:12.184492 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20445
2026-09-23T20:22:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3890 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:22:12 dist-test-slave-wmd0 krb5kdc[20960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:22:12 dist-test-slave-wmd0 krb5kdc[20960](info): set up 2 sockets
Sep 23 20:22:12 dist-test-slave-wmd0 krb5kdc[20960](info): commencing operation
krb5kdc: starting...
W20260923 20:22:13.264040 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:22:13 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.254.keytab.
Entry for principal oryx/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.254.keytab.
I20260923 20:22:13.474351 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46857
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.254.keytab
--principal=oryx/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:46857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:13.789242 20973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:13.789526 20973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:13.789726 20973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:13.796859 20973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:13.796958 20973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:13.797080 20973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:13.797179 20973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:13.797257 20973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:13.806926 20973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:46857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.254.keytab
--principal=oryx/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:13.808212 20973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:13.809501 20973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:13.821040 20978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:13.821003 20979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:13.820997 20981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:13.822422 20973 server_base.cc:1034] running on GCE node
I20260923 20:22:13.823215 20973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:13.824532 20973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:13.825726 20973 hybrid_clock.cc:648] HybridClock initialized: now 1790194933825682 us; error 41 us; skew 500 ppm
I20260923 20:22:13.826077 20973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:13 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:13.833290 20973 init.cc:377] Logged in from keytab as oryx/127.14.127.254@KRBTEST.COM (short username oryx)
I20260923 20:22:13.834892 20973 webserver.cc:460] Webserver started at http://127.14.127.254:41875/ using document root <none> and password file <none>
I20260923 20:22:13.835584 20973 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:13.835714 20973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:13.836025 20973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:13.839030 20973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "bdd278a282f14dc69e4d6c04c9f016a0"
format_stamp: "Formatted at 2026-09-23 20:22:13 on dist-test-slave-wmd0"
server_key: "20679541a7b610025a3703cb9d923cb7"
server_key_iv: "26fb0a5c310c8cb44b9f300f4ea05a8f"
server_key_version: "encryptionkey@0"
I20260923 20:22:13.839792 20973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "bdd278a282f14dc69e4d6c04c9f016a0"
format_stamp: "Formatted at 2026-09-23 20:22:13 on dist-test-slave-wmd0"
server_key: "20679541a7b610025a3703cb9d923cb7"
server_key_iv: "26fb0a5c310c8cb44b9f300f4ea05a8f"
server_key_version: "encryptionkey@0"
I20260923 20:22:13.844800 20973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:22:13.848155 20988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:13.849465 20973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:22:13.849666 20973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "bdd278a282f14dc69e4d6c04c9f016a0"
format_stamp: "Formatted at 2026-09-23 20:22:13 on dist-test-slave-wmd0"
server_key: "20679541a7b610025a3703cb9d923cb7"
server_key_iv: "26fb0a5c310c8cb44b9f300f4ea05a8f"
server_key_version: "encryptionkey@0"
I20260923 20:22:13.849857 20973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:13.861356 20973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:13.865336 20973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:13.865615 20973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:13.883761 20973 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:46857
I20260923 20:22:13.883800 21039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:46857 every 8 connection(s)
I20260923 20:22:13.885138 20973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:13.888717 21040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:13.893898 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 20973
I20260923 20:22:13.894109 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:13.894821 14847 external_mini_cluster.cc:1428] Setting key 0a4dbf6b8d9c3a28701d29e1b7b8169d
I20260923 20:22:13.899175 21040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: Bootstrap starting.
I20260923 20:22:13.902840 21040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:13.904035 21040 log.cc:824] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:13.907246 21040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: No bootstrap required, opened a new log
Sep 23 20:22:13 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:13.917776 21040 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } }
I20260923 20:22:13.918218 21040 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:13.918340 21040 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd278a282f14dc69e4d6c04c9f016a0, State: Initialized, Role: FOLLOWER
I20260923 20:22:13.918960 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:13.896739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.14.127.254:46857)
0923 20:22:13.897545 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:13.897556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:13.897595 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:22:13.899205 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:13.899226 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:13.899632 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:13.899935 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:13.901819 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:13.903228 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:13.904916 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:13.905744 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:13.905866 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:13.910410 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:13.910422 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:13.910446 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:13.910494 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:13.914460 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:13.915219 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:13.915228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:13.915238 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:13.915474 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:13.915790 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:13.915795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:13.915799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:13.916532 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:13.916834 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:13.917104 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 20:22:13.917570 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":403,"threads_started":1}
I20260923 20:22:13.919036 21040 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } }
I20260923 20:22:13.919301 21040 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:13.919477 21040 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:13.919667 21040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:13.921933 21040 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } }
I20260923 20:22:13.922411 21040 leader_election.cc:302] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdd278a282f14dc69e4d6c04c9f016a0; no voters:
I20260923 20:22:13.923007 21040 leader_election.cc:288] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:13.923146 21045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:13.923822 21045 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 1 LEADER]: Becoming Leader. State: Replica: bdd278a282f14dc69e4d6c04c9f016a0, State: Running, Role: LEADER
I20260923 20:22:13.924363 21045 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } }
I20260923 20:22:13.925017 21040 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:13.927101 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd278a282f14dc69e4d6c04c9f016a0. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } }
I20260923 20:22:13.927780 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:13.927456 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } }
I20260923 20:22:13.927923 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:13.928750 21054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:13.933621 21054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:13.943775 21054 catalog_manager.cc:1347] Generated new cluster ID: 1f06babdafee4507aef3d034fc0b8ee1
I20260923 20:22:13.943879 21054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:13.965310 21054 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:13.967092 21054 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:13.980142 21054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: Generated new TSK 0
I20260923 20:22:13.981073 21054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab.
Entry for principal oryx/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab.
I20260923 20:22:14.047526 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46857
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab
--principal=oryx/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:14.367336 21066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:14.367641 21066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:14.367811 21066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:14.374749 21066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:14.374900 21066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:14.375088 21066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:14.384627 21066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab
--principal=oryx/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:14.385885 21066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:14.387202 21066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:14.400703 21072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:14.400748 21074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:14.400841 21071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:14.401438 21066 server_base.cc:1034] running on GCE node
I20260923 20:22:14.402040 21066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:14.403026 21066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:14.404270 21066 hybrid_clock.cc:648] HybridClock initialized: now 1790194934404201 us; error 41 us; skew 500 ppm
I20260923 20:22:14.404582 21066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:14 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:14.407629 21066 init.cc:377] Logged in from keytab as oryx/127.14.127.193@KRBTEST.COM (short username oryx)
I20260923 20:22:14.409027 21066 webserver.cc:460] Webserver started at http://127.14.127.193:42291/ using document root <none> and password file <none>
I20260923 20:22:14.409652 21066 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:14.409809 21066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:14.410107 21066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:14.412909 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "f71c2dde189346db80b51359c5319f6a"
format_stamp: "Formatted at 2026-09-23 20:22:14 on dist-test-slave-wmd0"
server_key: "3961ab0ded216b3b3d5d4ee1ca2913a4"
server_key_iv: "b1a82c1f5544046331c3703adb182e8f"
server_key_version: "encryptionkey@0"
I20260923 20:22:14.413640 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "f71c2dde189346db80b51359c5319f6a"
format_stamp: "Formatted at 2026-09-23 20:22:14 on dist-test-slave-wmd0"
server_key: "3961ab0ded216b3b3d5d4ee1ca2913a4"
server_key_iv: "b1a82c1f5544046331c3703adb182e8f"
server_key_version: "encryptionkey@0"
I20260923 20:22:14.418767 21066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260923 20:22:14.422039 21081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:14.423311 21066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:14.423492 21066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "f71c2dde189346db80b51359c5319f6a"
format_stamp: "Formatted at 2026-09-23 20:22:14 on dist-test-slave-wmd0"
server_key: "3961ab0ded216b3b3d5d4ee1ca2913a4"
server_key_iv: "b1a82c1f5544046331c3703adb182e8f"
server_key_version: "encryptionkey@0"
I20260923 20:22:14.423687 21066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:14.471091 21066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:14.475134 21066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:14.475436 21066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:14.476384 21066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:14.477741 21066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:14.477842 21066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:14.477990 21066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:14.478061 21066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:14.511385 21066 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:36551
I20260923 20:22:14.511440 21193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:36551 every 8 connection(s)
I20260923 20:22:14.512673 21066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:14.518805 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 21066
I20260923 20:22:14.519027 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:14.519834 14847 external_mini_cluster.cc:1428] Setting key 134b8127c70b4111177764cbe003398e
Sep 23 20:22:14 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:14.535276 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:14.515243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:47721 (local address 127.14.127.254:46857)
0923 20:22:14.515673 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:14.515677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:14.516630 (+ 953us) server_negotiation.cc:407] Connection header received
0923 20:22:14.518532 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:14.518536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:14.518670 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:14.518778 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:14.520807 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:14.521808 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:14.523971 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:14.524679 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:14.524830 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:14.529137 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:14.529142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:14.529146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:14.529179 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:14.531870 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:14.532801 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:14.532805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:14.532807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:14.532932 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:14.533414 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:14.533418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:14.533421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:14.533667 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:14.533774 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:14.534775 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0923 20:22:14.534952 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260923 20:22:14.536823 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:14.515843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.193:47721)
0923 20:22:14.516487 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:14.516525 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:14.518166 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:14.519072 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:14.519084 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:14.519820 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:14.520605 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:14.520633 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:14.522009 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:14.522015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:14.523824 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:14.523837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:14.525026 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:14.525035 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:14.525340 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:14.526102 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:14.526133 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:14.528883 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:14.532024 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:14.532036 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:14.532056 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:14.532683 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:14.533040 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:14.533047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:14.533054 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:14.533307 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:14.533797 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:14.533816 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:14.534309 (+ 493us) client_negotiation.cc:769] Sending connection context
0923 20:22:14.534670 (+ 361us) client_negotiation.cc:240] Negotiation successful
0923 20:22:14.535280 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":261,"threads_started":1}
I20260923 20:22:14.538699 21194 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46857
I20260923 20:22:14.539042 21194 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:14.539974 21194 heartbeater.cc:511] Master 127.14.127.254:46857 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:22:14.542292 21005 ts_manager.cc:194] Registered new tserver with Master: f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.194@KRBTEST.COM" created.
I20260923 20:22:14.544116 21005 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:47721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab.
Entry for principal oryx/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab.
I20260923 20:22:14.568188 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46857
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab
--principal=oryx/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:14.884562 21200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:14.884802 21200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:14.884964 21200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:14.892073 21200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:14.892168 21200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:14.892330 21200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:14.901691 21200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab
--principal=oryx/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:14.902940 21200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:14.904170 21200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:14.916716 21205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:14.916761 21206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:14.916677 21208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:14.920032 21200 server_base.cc:1034] running on GCE node
I20260923 20:22:14.920670 21200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:14.921716 21200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:14.922924 21200 hybrid_clock.cc:648] HybridClock initialized: now 1790194934922869 us; error 50 us; skew 500 ppm
I20260923 20:22:14.923190 21200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:14 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:14.926373 21200 init.cc:377] Logged in from keytab as oryx/127.14.127.194@KRBTEST.COM (short username oryx)
I20260923 20:22:14.927909 21200 webserver.cc:460] Webserver started at http://127.14.127.194:46515/ using document root <none> and password file <none>
I20260923 20:22:14.928606 21200 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:14.928714 21200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:14.928943 21200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:14.931813 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "92efd8ec3f3a47e792d179a8b06cf105"
format_stamp: "Formatted at 2026-09-23 20:22:14 on dist-test-slave-wmd0"
server_key: "51f72e91ebffc89a5b7963a353d4ba85"
server_key_iv: "d46e93533aaa7a0d59671e4467c3c279"
server_key_version: "encryptionkey@0"
I20260923 20:22:14.932520 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "92efd8ec3f3a47e792d179a8b06cf105"
format_stamp: "Formatted at 2026-09-23 20:22:14 on dist-test-slave-wmd0"
server_key: "51f72e91ebffc89a5b7963a353d4ba85"
server_key_iv: "d46e93533aaa7a0d59671e4467c3c279"
server_key_version: "encryptionkey@0"
I20260923 20:22:14.937510 21200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:22:14.940999 21215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:14.942807 21200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:22:14.942962 21200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "92efd8ec3f3a47e792d179a8b06cf105"
format_stamp: "Formatted at 2026-09-23 20:22:14 on dist-test-slave-wmd0"
server_key: "51f72e91ebffc89a5b7963a353d4ba85"
server_key_iv: "d46e93533aaa7a0d59671e4467c3c279"
server_key_version: "encryptionkey@0"
I20260923 20:22:14.943202 21200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:14.968935 21200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:14.972946 21200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:14.973246 21200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:14.974212 21200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:14.975590 21200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:14.975695 21200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:14.975843 21200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:14.975917 21200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:15.010154 21200 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:42733
I20260923 20:22:15.010197 21327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:42733 every 8 connection(s)
I20260923 20:22:15.011524 21200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:15.017784 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 21200
I20260923 20:22:15.017987 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:15.018431 14847 external_mini_cluster.cc:1428] Setting key 7bdd04bbc1d5e2b07153498979fe90af
Sep 23 20:22:15 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194934, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:15.032244 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:15.014056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:56625 (local address 127.14.127.254:46857)
0923 20:22:15.014285 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:15.014292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:15.015546 (+ 1254us) server_negotiation.cc:407] Connection header received
0923 20:22:15.017282 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:15.017287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:15.017357 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:15.017511 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:15.019283 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:15.020114 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:15.021732 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:15.022151 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:15.022282 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:15.025924 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:15.025928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:15.025932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:15.025964 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:15.029012 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:15.029948 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:15.029952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:15.029954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:15.030072 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:15.030544 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:15.030547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:15.030549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:15.030861 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:15.030970 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:15.031849 (+ 879us) server_negotiation.cc:299] Negotiation successful
0923 20:22:15.032012 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 20:22:15.033810 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:15.014685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.194:56625)
0923 20:22:15.015383 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:15.015428 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:15.017027 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:15.017780 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:15.017792 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:15.018396 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:15.019106 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:15.019130 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:15.020262 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:15.020267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:15.021569 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:15.021580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:15.022364 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:15.022371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:15.022597 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:15.023317 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:15.023339 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:15.025707 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:15.029144 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:15.029156 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:15.029174 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:15.029813 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:15.030179 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:15.030186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:15.030193 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:15.030432 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:15.030990 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:15.031000 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:15.031447 (+ 447us) client_negotiation.cc:769] Sending connection context
0923 20:22:15.031769 (+ 322us) client_negotiation.cc:240] Negotiation successful
0923 20:22:15.032325 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":272,"threads_started":1}
I20260923 20:22:15.035486 21328 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46857
I20260923 20:22:15.035923 21328 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:15.036677 21328 heartbeater.cc:511] Master 127.14.127.254:46857 requested a full tablet report, sending...
I20260923 20:22:15.038175 21005 ts_manager.cc:194] Registered new tserver with Master: 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194:42733)
WARNING: no policy specified for oryx/127.14.127.195@KRBTEST.COM; defaulting to no policy
I20260923 20:22:15.039141 21005 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:56625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab.
Entry for principal oryx/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab.
I20260923 20:22:15.066382 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46857
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab
--principal=oryx/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:15.381114 21333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:15.381438 21333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:15.381608 21333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:15.388835 21333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:15.388988 21333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:15.389178 21333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:15.399073 21333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab
--principal=oryx/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:15.400326 21333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:15.401620 21333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:15.415234 21341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:15.415326 21338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:15.415335 21339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:15.415678 21333 server_base.cc:1034] running on GCE node
I20260923 20:22:15.416630 21333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:15.417580 21333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:15.418788 21333 hybrid_clock.cc:648] HybridClock initialized: now 1790194935418754 us; error 33 us; skew 500 ppm
I20260923 20:22:15.419132 21333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:15 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:15.422286 21333 init.cc:377] Logged in from keytab as oryx/127.14.127.195@KRBTEST.COM (short username oryx)
I20260923 20:22:15.423705 21333 webserver.cc:460] Webserver started at http://127.14.127.195:34683/ using document root <none> and password file <none>
I20260923 20:22:15.424350 21333 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:15.424450 21333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:15.424708 21333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:15.427831 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "a27f9793e4a744d1a135d602dec9ee6e"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "495369ff110cd9d3b326fd577f68acd0"
server_key_iv: "67f55f9cbc4f39cbece6a8c54e17afa1"
server_key_version: "encryptionkey@0"
I20260923 20:22:15.428640 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "a27f9793e4a744d1a135d602dec9ee6e"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "495369ff110cd9d3b326fd577f68acd0"
server_key_iv: "67f55f9cbc4f39cbece6a8c54e17afa1"
server_key_version: "encryptionkey@0"
I20260923 20:22:15.433634 21333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:22:15.437216 21348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:15.438560 21333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:22:15.438774 21333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "a27f9793e4a744d1a135d602dec9ee6e"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "495369ff110cd9d3b326fd577f68acd0"
server_key_iv: "67f55f9cbc4f39cbece6a8c54e17afa1"
server_key_version: "encryptionkey@0"
I20260923 20:22:15.438989 21333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:15.468371 21333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:15.473013 21333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:15.473318 21333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:15.474287 21333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:15.475664 21333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:15.475775 21333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:15.475924 21333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:15.475997 21333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:15.509692 21333 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:39771
I20260923 20:22:15.509739 21460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:39771 every 8 connection(s)
I20260923 20:22:15.511096 21333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:15.515770 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 21333
I20260923 20:22:15.515964 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:15.516414 14847 external_mini_cluster.cc:1428] Setting key 637943d53b26f3f9990cd77d554286fa
Sep 23 20:22:15 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:15.531685 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:15.513892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:42979 (local address 127.14.127.254:46857)
0923 20:22:15.514106 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:15.514113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:15.515442 (+ 1329us) server_negotiation.cc:407] Connection header received
0923 20:22:15.517566 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:15.517573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:15.517666 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:15.517776 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:15.519261 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:15.520023 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:15.521554 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:15.521972 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:15.522100 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:15.525768 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:15.525773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:15.525777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:15.525815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:15.528523 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:15.529301 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:15.529311 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:15.529314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:15.529458 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:15.529908 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:15.529918 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:15.529920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:15.530190 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:15.530349 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:15.531095 (+ 746us) server_negotiation.cc:299] Negotiation successful
0923 20:22:15.531319 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:22:15.532601 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:15.514425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.195:42979)
0923 20:22:15.515274 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:15.515335 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:15.517288 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:15.518001 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:15.518009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:15.518497 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:15.519098 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:15.519115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:15.520181 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:15.520185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:15.521400 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:15.521411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:15.522196 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:15.522203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:15.522445 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:15.523131 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:15.523154 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:15.525523 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:15.528670 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:15.528706 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:15.528722 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:15.529163 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:15.529576 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:15.529581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:15.529584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:15.529780 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:15.530320 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:15.530328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:15.530725 (+ 397us) client_negotiation.cc:769] Sending connection context
0923 20:22:15.530997 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 20:22:15.531523 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":370,"threads_started":1}
I20260923 20:22:15.533931 21461 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46857
I20260923 20:22:15.534191 21461 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:15.534989 21461 heartbeater.cc:511] Master 127.14.127.254:46857 requested a full tablet report, sending...
I20260923 20:22:15.536394 21005 ts_manager.cc:194] Registered new tserver with Master: a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195:39771)
I20260923 20:22:15.537210 21005 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:42979
I20260923 20:22:15.543767 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:22:15.546900 21194 heartbeater.cc:503] Master 127.14.127.254:46857 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.14.127.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.253.keytab.
Entry for principal oryx/127.14.127.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.253.keytab.
I20260923 20:22:15.590298 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.253:34901
--webserver_interface=127.14.127.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.253.keytab
--principal=oryx/127.14.127.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:46857,127.14.127.253:34901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:15.916116 21466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:15.916456 21466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:15.916610 21466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:15.923898 21466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:15.924041 21466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:15.924142 21466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:15.924245 21466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:15.924320 21466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:15.934378 21466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:46857,127.14.127.253:34901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.253.keytab
--principal=oryx/127.14.127.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.253:34901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.127.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
W20260923 20:22:15.934785 21466 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.127.254:46857,127.14.127.253:34901'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 20:22:15.935858 21466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:15.937337 21466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:15.949483 21471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:15.949502 21472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:15.949529 21474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:15.950438 21466 server_base.cc:1034] running on GCE node
I20260923 20:22:15.951228 21466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:15.952548 21466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:15.953768 21466 hybrid_clock.cc:648] HybridClock initialized: now 1790194935953701 us; error 56 us; skew 500 ppm
I20260923 20:22:15.954244 21466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:15 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:15.957549 21466 init.cc:377] Logged in from keytab as oryx/127.14.127.253@KRBTEST.COM (short username oryx)
I20260923 20:22:15.959036 21466 webserver.cc:460] Webserver started at http://127.14.127.253:40671/ using document root <none> and password file <none>
I20260923 20:22:15.959769 21466 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:15.959859 21466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:15.960182 21466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:15.963163 21466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data/instance:
uuid: "dff60975532f4b29868d003ed8fd3f92"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "d326c294f1f7f0c6ec8c5ba570a7d461"
server_key_iv: "25c76ddefbb4e8d10117fa74f8eeb352"
server_key_version: "encryptionkey@0"
I20260923 20:22:15.963924 21466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal/instance:
uuid: "dff60975532f4b29868d003ed8fd3f92"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "d326c294f1f7f0c6ec8c5ba570a7d461"
server_key_iv: "25c76ddefbb4e8d10117fa74f8eeb352"
server_key_version: "encryptionkey@0"
I20260923 20:22:15.969189 21466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260923 20:22:15.972525 21481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:15.973901 21466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:22:15.974076 21466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
uuid: "dff60975532f4b29868d003ed8fd3f92"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "d326c294f1f7f0c6ec8c5ba570a7d461"
server_key_iv: "25c76ddefbb4e8d10117fa74f8eeb352"
server_key_version: "encryptionkey@0"
I20260923 20:22:15.974291 21466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:15.989306 21466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:15.993283 21466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:15.993556 21466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:16.012382 21466 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.253:34901
I20260923 20:22:16.012401 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.253:34901 every 8 connection(s)
I20260923 20:22:16.013727 21466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
I20260923 20:22:16.019033 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 21466
I20260923 20:22:16.020481 21533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:16.031332 21533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.253@KRBTEST.COM
I20260923 20:22:16.042143 21328 heartbeater.cc:503] Master 127.14.127.254:46857 was elected leader, sending a full tablet report...
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.253@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:16.048007 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.021392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.14.127.253:34901)
0923 20:22:16.022237 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.022254 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.027282 (+ 5028us) server_negotiation.cc:407] Connection header received
0923 20:22:16.029019 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.029043 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.029458 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.029842 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.031401 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.032837 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.034265 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.035240 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.035406 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.040227 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.040237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.040258 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.040296 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.043998 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.044681 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.044696 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.044703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.044849 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.045253 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.045264 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.045266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.045754 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.046059 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.046517 (+ 458us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.046986 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":339,"threads_started":1}
I20260923 20:22:16.049911 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.033908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.14.127.254:46857)
0923 20:22:16.034380 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.034386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.034991 (+ 605us) server_negotiation.cc:407] Connection header received
0923 20:22:16.035168 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.035174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.035264 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.035378 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.036770 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.037487 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.039229 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.039914 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.040078 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.044114 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.044120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.044125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.044167 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.047106 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.047817 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.047822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.047826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.047961 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.048422 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.048427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.048430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.048728 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.048882 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.049333 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.049552 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260923 20:22:16.049971 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.033942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:44926)
0923 20:22:16.034844 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.034884 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.035041 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.035462 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.035470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.036268 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.036607 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.036622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.037624 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.037631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.039064 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.039096 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.040133 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.040138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.040267 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.040956 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.040984 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.043934 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.047227 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.047239 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.047246 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.047691 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.048118 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.048125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.048130 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.048313 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.048859 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.048866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.049045 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.049242 (+ 197us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.049547 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":480,"threads_started":1}
I20260923 20:22:16.051724 21533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194935, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.253@KRBTEST.COM for oryx/127.14.127.253@KRBTEST.COM
I20260923 20:22:16.067601 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.052536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.0.0.1:48790)
0923 20:22:16.052689 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.052704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.052785 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.053065 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.053068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.053353 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.053579 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.053591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.054657 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.054662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.056284 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.056302 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.057394 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.057403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.057641 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.058262 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.058290 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.061366 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.065182 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.065185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.065188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.065541 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.065893 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.065896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.065898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.066021 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.066581 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.066584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.066745 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.066932 (+ 187us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.067235 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:22:16.067731 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.052422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.14.127.253:34901)
0923 20:22:16.052583 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.052592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.052777 (+ 185us) server_negotiation.cc:407] Connection header received
0923 20:22:16.052853 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.052856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.052939 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.053036 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.053713 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.054513 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.056473 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.057176 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.057307 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.061548 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.061553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.061556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.061583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.065046 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.065645 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.065650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.065653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.065789 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.066124 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.066129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.066133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.066469 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.066656 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.067029 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.067396 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":56}
I20260923 20:22:16.072388 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Bootstrap starting.
I20260923 20:22:16.075387 21533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:16.076364 21533 log.cc:824] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:16.079192 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: No bootstrap required, opened a new log
I20260923 20:22:16.085879 21533 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } }
I20260923 20:22:16.086174 21533 raft_consensus.cc:374] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:16.086289 21533 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff60975532f4b29868d003ed8fd3f92, State: Initialized, Role: FOLLOWER
I20260923 20:22:16.086902 21533 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } }
I20260923 20:22:16.087850 21540 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } } }
I20260923 20:22:16.088203 21540 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:22:16.088930 21533 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:22:16.098636 21551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:22:16.098779 21551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:22:16.101202 21466 master_runner.cc:428] Detected that this master dff60975532f4b29868d003ed8fd3f92 is joining an existing cluster
I20260923 20:22:16.101315 21466 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.127.253:34901
WARNING: no policy specified for oryx/127.14.127.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.252@KRBTEST.COM" created.
I20260923 20:22:16.164147 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.130888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.0.0.1:48806)
0923 20:22:16.136374 (+ 5486us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.136398 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.136574 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.143943 (+ 7369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.143951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.145662 (+ 1711us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.146041 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.146057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.147658 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.147663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.149371 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.149386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.150671 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.150678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.150855 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.151563 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.151590 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.152871 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.157091 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.157097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.157103 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.157732 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.159593 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.159600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.159606 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.159775 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.161437 (+ 1662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.161444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.161627 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.161775 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.163491 (+ 1716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4511,"mutex_wait_us":30,"thread_start_us":4338,"threads_started":1}
I20260923 20:22:16.165820 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.130722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.14.127.253:34901)
0923 20:22:16.130936 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.130955 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.136494 (+ 5539us) server_negotiation.cc:407] Connection header received
0923 20:22:16.136726 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.136732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.136820 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.136928 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.146217 (+ 9289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.147395 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.149542 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.150269 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.150383 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.153064 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.153070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.153075 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.153118 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.156905 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.159267 (+ 2362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.159275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.159281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.159459 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.160732 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.160739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.160745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.161183 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.161355 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.165064 (+ 3709us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.165342 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:22:16.172102 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.133807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.14.127.254:46857)
0923 20:22:16.134018 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.134026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.143271 (+ 9245us) server_negotiation.cc:407] Connection header received
0923 20:22:16.143553 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.143562 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.143661 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.143743 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.145225 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.146587 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.149333 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.150219 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.150345 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.154979 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.154988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.154996 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.155044 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.160539 (+ 5495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.169224 (+ 8685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.169232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.169238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.169440 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.169951 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.169958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.169965 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.170396 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.170553 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.171263 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.171650 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:22:16.173502 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.133787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:44932)
0923 20:22:16.143149 (+ 9362us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.143170 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.143380 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.143976 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.143981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.144684 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.145028 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.145044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.146827 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.146835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.149126 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.149148 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.150789 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.150798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.151039 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.153242 (+ 2203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.153271 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.154728 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.168455 (+ 13727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.168463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.168469 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.169055 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.169616 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.169623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.169632 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.169814 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.170663 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.170671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.170904 (+ 233us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.172513 (+ 1609us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.172925 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9197,"mutex_wait_us":29,"thread_start_us":9832,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.252.keytab.
Entry for principal oryx/127.14.127.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.252.keytab.
I20260923 20:22:16.179965 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.252:43217
--webserver_interface=127.14.127.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.252.keytab
--principal=oryx/127.14.127.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:46857,127.14.127.253:34901,127.14.127.252:43217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.254@KRBTEST.COM for oryx/127.14.127.253@KRBTEST.COM
I20260923 20:22:16.213472 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.186894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.14.127.253:34901)
0923 20:22:16.187063 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.187069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.195285 (+ 8216us) server_negotiation.cc:407] Connection header received
0923 20:22:16.195675 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.195681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.195801 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.195892 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.197588 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.198734 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.201071 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.201800 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.201893 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.206324 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.206331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.206336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.206374 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.210149 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.211114 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.211120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.211124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.211279 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.211718 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.211723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.211727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.212076 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.212204 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.212841 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.213096 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:22:16.214493 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.186556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.0.0.1:48814)
0923 20:22:16.195135 (+ 8579us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.195174 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.195428 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.196194 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.196202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.197099 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.197408 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.197422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.199262 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.199268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.200896 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.200911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.202212 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.202218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.202407 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.203185 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.203214 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.206121 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.210336 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.210352 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.210361 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.210979 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.211425 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.211430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.211434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.211589 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.212296 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.212304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.212498 (+ 194us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.213731 (+ 1233us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.214083 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8327,"thread_start_us":8177,"threads_started":1}
I20260923 20:22:16.216671 21005 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.127.253:34901: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dff60975532f4b29868d003ed8fd3f92"
member_type: NON_VOTER
last_known_addr {
host: "127.14.127.253"
port: 34901
}
attrs {
promote: true
}
}
dest_uuid: "bdd278a282f14dc69e4d6c04c9f016a0"
cas_config_opid_index: -1
I20260923 20:22:16.218056 21005 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } }
I20260923 20:22:16.235042 21564 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dff60975532f4b29868d003ed8fd3f92 (127.14.127.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:16.243227 21508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 0 FOLLOWER]: Advancing to term 1
W20260923 20:22:16.245672 21487 tablet.cc:2369] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:22:16.247076 21508 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Refusing update from remote peer bdd278a282f14dc69e4d6c04c9f016a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 20:22:16.251001 21566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:16.252830 21564 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.127.253:34901
I20260923 20:22:16.258980 21466 master.cc:537] Master@127.14.127.253:34901 shutting down...
I20260923 20:22:16.259063 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:16.259508 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:16.318138 21508 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dff60975532f4b29868d003ed8fd3f92 (127.14.127.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:16.327476 21540 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd278a282f14dc69e4d6c04c9f016a0. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:16.327744 21540 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:22:16.328088 21570 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:16.328403 21570 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:22:16.330685 21574 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:16.330937 21574 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:22:16.331784 21466 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 LEARNER]: Raft consensus shutting down.
I20260923 20:22:16.332252 21466 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 LEARNER]: Raft consensus is shut down!
I20260923 20:22:16.332415 21466 tablet_replica.cc:330] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: stopping tablet replica
I20260923 20:22:16.351012 21466 master.cc:552] Master@127.14.127.253:34901 shutdown complete.
I20260923 20:22:16.351112 21466 master_runner.cc:305] Clearing existing system tablet
I20260923 20:22:16.351545 21466 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:16.351637 21466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:16.361619 21466 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.006s sys 0.004s
I20260923 20:22:16.371191 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:16.371785 21466 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:16.372061 21466 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260923 20:22:16.372151 21466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
uuid: "dff60975532f4b29868d003ed8fd3f92"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "d326c294f1f7f0c6ec8c5ba570a7d461"
server_key_iv: "25c76ddefbb4e8d10117fa74f8eeb352"
server_key_version: "encryptionkey@0"
I20260923 20:22:16.377013 21466 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:16.384037 21466 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 20:22:16.384218 21466 log.cc:1196] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 20:22:16.384724 21466 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Deleting consensus metadata
I20260923 20:22:16.385000 21466 master_runner.cc:315] Copying system tablet from 127.14.127.254:46857
I20260923 20:22:16.387749 21466 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: tablet copy: Beginning tablet copy session from remote peer at address 127.14.127.254:46857
I20260923 20:22:16.405184 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.389274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.14.127.254:46857)
0923 20:22:16.389505 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.389513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.389967 (+ 454us) server_negotiation.cc:407] Connection header received
0923 20:22:16.390200 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.390206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.390303 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.390417 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.391770 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.392978 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.394851 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.395576 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.395708 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.397987 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.397994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.397999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.398039 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.401788 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.402911 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.402919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.402925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.403118 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.403577 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.403581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.403585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.403952 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.404097 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.404580 (+ 483us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.404800 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 20:22:16.405265 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.389176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:44948)
0923 20:22:16.389813 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.389839 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.390038 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.390486 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.390493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.391209 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.391564 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.391607 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.393166 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.393172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.394701 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.394715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.395759 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.395765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.395944 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.396686 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.396709 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.397781 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.401959 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.401967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.401974 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.402708 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.403257 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.403263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.403267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.403432 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.404075 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.404080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.404264 (+ 184us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.404424 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.404752 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":344,"threads_started":1}
I20260923 20:22:16.406672 21025 tablet_copy_service.cc:140] P bdd278a282f14dc69e4d6c04c9f016a0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dff60975532f4b29868d003ed8fd3f92 ({username='oryx', principal='oryx/127.14.127.253@KRBTEST.COM'} at 127.0.0.1:44948)
I20260923 20:22:16.407153 21025 tablet_copy_service.cc:161] P bdd278a282f14dc69e4d6c04c9f016a0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dff60975532f4b29868d003ed8fd3f92 at {username='oryx', principal='oryx/127.14.127.253@KRBTEST.COM'} at 127.0.0.1:44948: session id = dff60975532f4b29868d003ed8fd3f92-00000000000000000000000000000000
I20260923 20:22:16.412289 21025 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 20:22:16.414916 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 20:22:16.422545 21466 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: tablet copy: Starting download of 0 data blocks...
I20260923 20:22:16.422969 21466 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: tablet copy: Starting download of 1 WAL segments...
I20260923 20:22:16.426370 21466 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 20:22:16.431484 21025 tablet_copy_service.cc:342] P bdd278a282f14dc69e4d6c04c9f016a0: Request end of tablet copy session dff60975532f4b29868d003ed8fd3f92-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.127.253@KRBTEST.COM'} at 127.0.0.1:44948
I20260923 20:22:16.431769 21025 tablet_copy_service.cc:433] P bdd278a282f14dc69e4d6c04c9f016a0: ending tablet copy session dff60975532f4b29868d003ed8fd3f92-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dff60975532f4b29868d003ed8fd3f92
I20260923 20:22:16.444591 21466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:16.447682 21583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:16.447867 21584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:16.455236 21586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:16.455796 21466 server_base.cc:1034] running on GCE node
I20260923 20:22:16.456200 21466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:16.456956 21466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:16.458268 21466 hybrid_clock.cc:648] HybridClock initialized: now 1790194936458203 us; error 34 us; skew 500 ppm
I20260923 20:22:16.458530 21466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:16.460882 21466 init.cc:377] Logged in from keytab as oryx/127.14.127.253@KRBTEST.COM (short username oryx)
I20260923 20:22:16.463613 21466 webserver.cc:460] Webserver started at http://127.14.127.253:34589/ using document root <none> and password file <none>
I20260923 20:22:16.463994 21466 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:16.464087 21466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:16.467622 21466 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 20:22:16.469698 21593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:16.470155 21466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 20:22:16.470332 21466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
uuid: "dff60975532f4b29868d003ed8fd3f92"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "d326c294f1f7f0c6ec8c5ba570a7d461"
server_key_iv: "25c76ddefbb4e8d10117fa74f8eeb352"
server_key_version: "encryptionkey@0"
I20260923 20:22:16.470561 21466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:16.493700 21466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:16.498081 21466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:16.527853 21466 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.253:34901
I20260923 20:22:16.527899 21644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.253:34901 every 8 connection(s)
I20260923 20:22:16.529165 21466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
I20260923 20:22:16.534684 21645 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:22:16.538779 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Bootstrap starting.
I20260923 20:22:16.539709 21461 heartbeater.cc:503] Master 127.14.127.254:46857 was elected leader, sending a full tablet report...
I20260923 20:22:16.560758 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:22:16.561342 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Bootstrap complete.
I20260923 20:22:16.562547 21645 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } }
I20260923 20:22:16.562851 21645 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dff60975532f4b29868d003ed8fd3f92, State: Initialized, Role: LEARNER
I20260923 20:22:16.563112 21645 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } }
I20260923 20:22:16.563988 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:16.564301 21645 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:16.564483 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:22:16.573549 21659 catalog_manager.cc:1259] Loaded cluster ID: 1f06babdafee4507aef3d034fc0b8ee1
I20260923 20:22:16.573668 21659 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: loading cluster ID for follower catalog manager: success
I20260923 20:22:16.578329 21659 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: acquiring CA information for follower catalog manager: success
I20260923 20:22:16.581838 21659 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 20:22:16.655666 21561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:16.655977 21561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:16.656152 21561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:16.664049 21561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:16.664191 21561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:16.664258 21561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260923 20:22:16.664077 21564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
W20260923 20:22:16.664330 21561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:16.664392 21561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:16.664392 21564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:16.675763 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.664451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.0.0.1:48826)
0923 20:22:16.664709 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.664731 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.664859 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.666511 (+ 1652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.666517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.666550 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:16.666993 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.667009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.669276 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.669282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.672219 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.672236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.674793 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.674798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.675227 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.675272 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.675375 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.675486 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260923 20:22:16.676241 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.665115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.14.127.253:34901)
0923 20:22:16.666041 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.666047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.666070 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:22:16.666143 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.666148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.666236 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.666411 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:16.667184 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.669047 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.672427 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.674438 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.674780 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.675022 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.675432 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.675597 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":584,"threads_started":1}
I20260923 20:22:16.677824 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd278a282f14dc69e4d6c04c9f016a0. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:16.678217 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:22:16.678848 21566 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: attempting to promote NON_VOTER dff60975532f4b29868d003ed8fd3f92 to VOTER
I20260923 20:22:16.679714 21566 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } }
I20260923 20:22:16.681257 21561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:46857,127.14.127.253:34901,127.14.127.252:43217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.252.keytab
--principal=oryx/127.14.127.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.252:43217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.127.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:16.681690 21620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 LEARNER]: Refusing update from remote peer bdd278a282f14dc69e4d6c04c9f016a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 20:22:16.682389 21564 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 20:22:16.683130 21561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:16.684521 21561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 20:22:16.686478 21566 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dff60975532f4b29868d003ed8fd3f92 (127.14.127.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } }
I20260923 20:22:16.687592 21620 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dff60975532f4b29868d003ed8fd3f92 (127.14.127.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } }
I20260923 20:22:16.690999 21564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } }
I20260923 20:22:16.691244 21564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:16.691581 21564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } }
I20260923 20:22:16.691811 21564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:16.691883 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } }
I20260923 20:22:16.692157 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 20:22:16.699671 21666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:16.699713 21669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:16.699630 21667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:16.700352 21561 server_base.cc:1034] running on GCE node
I20260923 20:22:16.701169 21561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:16.702677 21561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:16.703917 21561 hybrid_clock.cc:648] HybridClock initialized: now 1790194936703867 us; error 59 us; skew 500 ppm
I20260923 20:22:16.704270 21561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:16.707412 21561 init.cc:377] Logged in from keytab as oryx/127.14.127.252@KRBTEST.COM (short username oryx)
I20260923 20:22:16.709002 21561 webserver.cc:460] Webserver started at http://127.14.127.252:44835/ using document root <none> and password file <none>
I20260923 20:22:16.709635 21561 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:16.709731 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:16.709976 21561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:16.712841 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data/instance:
uuid: "cf74fd686b9a4dbbb3937086a532fd9b"
format_stamp: "Formatted at 2026-09-23 20:22:16 on dist-test-slave-wmd0"
server_key: "a857d305188946b15cb9b2d1a73c7381"
server_key_iv: "47ef1608bcf30990228f66083fd8f60f"
server_key_version: "encryptionkey@0"
I20260923 20:22:16.713536 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal/instance:
uuid: "cf74fd686b9a4dbbb3937086a532fd9b"
format_stamp: "Formatted at 2026-09-23 20:22:16 on dist-test-slave-wmd0"
server_key: "a857d305188946b15cb9b2d1a73c7381"
server_key_iv: "47ef1608bcf30990228f66083fd8f60f"
server_key_version: "encryptionkey@0"
I20260923 20:22:16.718767 21561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260923 20:22:16.722031 21676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:16.723417 21561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:16.723611 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
uuid: "cf74fd686b9a4dbbb3937086a532fd9b"
format_stamp: "Formatted at 2026-09-23 20:22:16 on dist-test-slave-wmd0"
server_key: "a857d305188946b15cb9b2d1a73c7381"
server_key_iv: "47ef1608bcf30990228f66083fd8f60f"
server_key_version: "encryptionkey@0"
I20260923 20:22:16.723794 21561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:16.746727 21561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:16.751219 21561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:16.751510 21561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:16.771979 21561 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.252:43217
I20260923 20:22:16.772065 21727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.252:43217 every 8 connection(s)
I20260923 20:22:16.773406 21561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
I20260923 20:22:16.777179 21728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:16.779682 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 21561
I20260923 20:22:16.784490 21728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.252@KRBTEST.COM
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.252@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:16.806030 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.787611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.14.127.254:46857)
0923 20:22:16.787772 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.787776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.788475 (+ 699us) server_negotiation.cc:407] Connection header received
0923 20:22:16.788710 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.788714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.788782 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.788900 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.790363 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.791470 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.793446 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.793894 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.794003 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.799483 (+ 5480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.799488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.799491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.799526 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.803215 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.803957 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.803962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.803966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.804104 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.804484 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.804491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.804494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.804793 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.804947 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.805430 (+ 483us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.805659 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:22:16.807109 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.781830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.14.127.252:43217)
0923 20:22:16.782728 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.782740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.782780 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:22:16.784401 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.784421 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.784813 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.785161 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.786991 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.789157 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.790912 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.791918 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.792058 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.798594 (+ 6536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.798603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.798663 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.798706 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.802676 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.803417 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.803423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.803429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.803584 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.803986 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.803990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.803992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.804527 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.804840 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.805266 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.805870 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":213,"server-negotiator.queue_time_us":632,"thread_start_us":402,"threads_started":1}
I20260923 20:22:16.807128 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.787627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:44958)
0923 20:22:16.788321 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.788376 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.788553 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.788958 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.788965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.789818 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.790150 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.790164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.791633 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.791637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.793261 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.793278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.794183 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.794194 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.794449 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.795318 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.795363 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.799247 (+ 3884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.803356 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.803368 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.803387 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.803831 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.804217 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.804223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.804226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.804372 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.804923 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.804931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.805124 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.805335 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.806022 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":112,"thread_start_us":371,"threads_started":1}
I20260923 20:22:16.808650 21728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.252@KRBTEST.COM for oryx/127.14.127.253@KRBTEST.COM
I20260923 20:22:16.821262 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.809478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.14.127.253:34901)
0923 20:22:16.809641 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.809647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.809764 (+ 117us) server_negotiation.cc:407] Connection header received
0923 20:22:16.809950 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.809954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.810017 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.810121 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.810823 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.811598 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.812977 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.813390 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.813520 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.816452 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.816457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.816460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.816493 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.819028 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.819695 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.819699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.819701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.819833 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.820178 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.820181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.820183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.820444 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.820551 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.820870 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.821042 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 20:22:16.821293 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.809438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.0.0.1:48840)
0923 20:22:16.809676 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.809693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.809803 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.810139 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.810144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.810496 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.810675 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.810686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.811848 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.811856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.812817 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.812828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.813537 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.813543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.813735 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.814243 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.814260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.816270 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.819168 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.819171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.819174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.819588 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.819950 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.819954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.819956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.820087 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.820541 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.820544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.820668 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.820775 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.820978 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260923 20:22:16.822443 21728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.252@KRBTEST.COM for oryx/127.14.127.252@KRBTEST.COM
I20260923 20:22:16.834918 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.823274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.14.127.252:43217)
0923 20:22:16.823408 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.823413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.823430 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:22:16.823543 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.823550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.823603 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.823697 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.824330 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.825003 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.826127 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.826516 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.826667 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.829724 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.829728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.829731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.829757 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.832298 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.832990 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.832998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.833001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.833119 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.833488 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.833495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.833498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.833779 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.833946 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.834255 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.834574 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":45}
I20260923 20:22:16.834914 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.823107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.0.0.1:43510)
0923 20:22:16.823247 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.823261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.823413 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.823710 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.823714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.824069 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.824205 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.824213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.825120 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.825127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.825999 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.826009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.826674 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.826679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.826975 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.827469 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.827485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.829543 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.832512 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.832519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.832523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.832859 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.833228 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.833235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.833238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.833385 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.833890 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.833903 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.834033 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.834184 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.834444 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":40}
I20260923 20:22:16.839641 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Bootstrap starting.
I20260923 20:22:16.842890 21728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:16.843966 21728 log.cc:824] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:16.846470 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: No bootstrap required, opened a new log
I20260923 20:22:16.853412 21728 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } }
I20260923 20:22:16.853696 21728 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:16.853843 21728 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf74fd686b9a4dbbb3937086a532fd9b, State: Initialized, Role: FOLLOWER
I20260923 20:22:16.854426 21728 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } }
I20260923 20:22:16.855234 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } } }
I20260923 20:22:16.855530 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:22:16.856230 21728 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:16.862118 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 21066
W20260923 20:22:16.865988 21745 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:22:16.866114 21745 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:22:16.869732 21561 master_runner.cc:428] Detected that this master cf74fd686b9a4dbbb3937086a532fd9b is joining an existing cluster
I20260923 20:22:16.869856 21561 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.127.252:43217
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.193@KRBTEST.COM
I20260923 20:22:16.888665 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.863047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.14.127.193:36551)
0923 20:22:16.866857 (+ 3810us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.866866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.866896 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:22:16.866969 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.866979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.867237 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.867435 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.869333 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.870925 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.873019 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.873871 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.874612 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.879912 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.879926 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.879942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.879985 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.884759 (+ 4774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.885565 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.885574 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.885577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.885707 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.886128 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.886133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.886136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.886528 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.887829 (+ 1301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.887996 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.888272 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3620,"thread_start_us":3424,"threads_started":1}
I20260923 20:22:16.890249 21189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:16.896438 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.874309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.14.127.253:34901)
0923 20:22:16.874857 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.874863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.875391 (+ 528us) server_negotiation.cc:407] Connection header received
0923 20:22:16.875652 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.875660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.875747 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.875818 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.877472 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.878379 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.880992 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.882320 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.882458 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.886791 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.886800 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.886808 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.886921 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.891080 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.892161 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.892169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.892175 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.892343 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.893459 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.893467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.893472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.893881 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.894039 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.895648 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.895929 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424}
I20260923 20:22:16.899623 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.874413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.0.0.1:48842)
0923 20:22:16.875286 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.875308 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.875501 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.876078 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.876085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.876807 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.877264 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.877279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.878524 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.878527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.880763 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.880783 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.882993 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.883002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.883388 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.884394 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.884425 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.886533 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.891310 (+ 4777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.891318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.891324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.892005 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.892500 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.892507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.892512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.892715 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.894132 (+ 1417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.894139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.894389 (+ 250us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.896762 (+ 2373us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.899100 (+ 2338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":767,"threads_started":1}
I20260923 20:22:16.915426 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.875618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.0.0.1:43518)
0923 20:22:16.893280 (+ 17662us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.893301 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.894706 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.895276 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.895284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.899738 (+ 4454us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.899998 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.900006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.901492 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.901498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.903167 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.903180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.904513 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.904518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.904682 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.905378 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.905420 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.906503 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.911930 (+ 5427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.911935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.911939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.912591 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.913083 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.913088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.913091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.913295 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.914055 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.914059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.914202 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.914361 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.914908 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17505,"mutex_wait_us":23,"thread_start_us":29487,"threads_started":1}
I20260923 20:22:16.916127 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.875884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.14.127.252:43217)
0923 20:22:16.876059 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.876065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.893373 (+ 17308us) server_negotiation.cc:407] Connection header received
0923 20:22:16.894874 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.894882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.894963 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.895044 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.900132 (+ 5088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.901276 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.903309 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.904084 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.904227 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.906691 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.906696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.906700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.906732 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.911768 (+ 5036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.912717 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.912722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.912725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.912960 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.913419 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.913424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.913427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.913855 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.914007 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.914797 (+ 790us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.915775 (+ 978us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":757,"server-negotiator.queue_time_us":51}
I20260923 20:22:16.924095 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.874995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.14.127.254:46857)
0923 20:22:16.875177 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.875182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.899314 (+ 24132us) server_negotiation.cc:407] Connection header received
0923 20:22:16.900390 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.900396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.900481 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.900554 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.902037 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.903167 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.906963 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.907810 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.907941 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.917366 (+ 9425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.917373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.917378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.917419 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.921119 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.921889 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.921897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.921901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.922057 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.922470 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.922475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.922479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.922852 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.923041 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.923512 (+ 471us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.923737 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:22:16.924103 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.882131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:44970)
0923 20:22:16.899206 (+ 17075us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.899230 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.900198 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.900746 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.900754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.901403 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.901844 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.901857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.903355 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.903372 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.905007 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.905021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.908407 (+ 3386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.908428 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.908660 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.909575 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.909602 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.917147 (+ 7545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.921265 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.921270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.921274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.921740 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.922174 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.922179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.922183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.922343 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.922986 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.922991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.923213 (+ 222us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.923351 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.923639 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16910,"mutex_wait_us":1,"thread_start_us":12226,"threads_started":1}
Sep 23 20:22:16 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.254@KRBTEST.COM for oryx/127.14.127.252@KRBTEST.COM
I20260923 20:22:16.945116 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.929301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.0.0.1:43520)
0923 20:22:16.929545 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:16.929563 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:16.929677 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:16.930127 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:16.930134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:16.930564 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:16.930791 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.930802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.932164 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.932170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.933575 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:16.933592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.934716 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.934722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:16.934920 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.935513 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:16.935537 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:16.938274 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:16.942364 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.942369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.942373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.942872 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.943316 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:16.943321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:16.943325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:16.943477 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:16.944075 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:16.944081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:16.944252 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 20:22:16.944406 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 20:22:16.944700 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260923 20:22:16.945123 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:16.929368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.14.127.252:43217)
0923 20:22:16.929518 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:16.929524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:16.929639 (+ 115us) server_negotiation.cc:407] Connection header received
0923 20:22:16.929869 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:16.929875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:16.929977 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:16.930067 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:16.930946 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.932021 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.933736 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:16.934524 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:16.934670 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:16.938474 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:16.938483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:16.938489 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:16.938524 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:16.942217 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.943029 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.943034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.943038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.943190 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:16.943596 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:16.943601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:16.943605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:16.943947 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:16.944082 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:16.944541 (+ 459us) server_negotiation.cc:299] Negotiation successful
0923 20:22:16.944771 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:22:16.947144 21005 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.127.252:43217: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b"
member_type: NON_VOTER
last_known_addr {
host: "127.14.127.252"
port: 43217
}
attrs {
promote: true
}
}
dest_uuid: "bdd278a282f14dc69e4d6c04c9f016a0"
cas_config_opid_index: 6
I20260923 20:22:16.947959 21005 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } }
I20260923 20:22:16.950390 21620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Refusing update from remote peer bdd278a282f14dc69e4d6c04c9f016a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 20:22:16.950538 21703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:16.951051 21564 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 20:22:16.953771 21564 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cf74fd686b9a4dbbb3937086a532fd9b (127.14.127.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:16.954936 21703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 FOLLOWER]: Refusing update from remote peer bdd278a282f14dc69e4d6c04c9f016a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 20:22:16.958173 21566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:16.960367 21620 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER cf74fd686b9a4dbbb3937086a532fd9b (127.14.127.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:16.964283 21564 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.127.252:43217
I20260923 20:22:16.965095 21561 master.cc:537] Master@127.14.127.252:43217 shutting down...
I20260923 20:22:16.966126 21566 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:16.966421 21566 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:16.966835 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:16.966998 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:16.967299 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:22:16.967290 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:16.980635 21703 raft_consensus.cc:858] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } }
W20260923 20:22:17.006932 21682 tablet.cc:2369] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:22:16.980966 21703 raft_consensus.cc:1565] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } }
I20260923 20:22:17.008747 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 21066
I20260923 20:22:17.026445 21703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dff60975532f4b29868d003ed8fd3f92 (127.14.127.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:17.031848 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:17.032227 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 20:22:17.032390 21758 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:17.032606 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd278a282f14dc69e4d6c04c9f016a0. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: NON_VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: true } } }
I20260923 20:22:17.032778 21758 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 20:22:17.032856 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260923 20:22:17.035007 21564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:17.035362 21564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:17.036067 21561 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260923 20:22:17.036375 20991 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260923 20:22:17.036535 21561 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260923 20:22:17.036700 21561 tablet_replica.cc:330] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: stopping tablet replica
I20260923 20:22:17.039863 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:36551
--local_ip_for_outbound_sockets=127.14.127.193
--tserver_master_addrs=127.14.127.254:46857
--webserver_port=42291
--webserver_interface=127.14.127.193
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab
--principal=oryx/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901,127.14.127.252:43217 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:22:17.047749 21561 master.cc:552] Master@127.14.127.252:43217 shutdown complete.
I20260923 20:22:17.047843 21561 master_runner.cc:305] Clearing existing system tablet
I20260923 20:22:17.048244 21561 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:17.048245 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.047748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.0.0.1:43522)
0923 20:22:17.047938 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:17.048085 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:43217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 20:22:17.048355 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260923 20:22:17.048458 20991 consensus_peers.cc:487] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 -> Peer cf74fd686b9a4dbbb3937086a532fd9b (127.14.127.252:43217): Couldn't send request to peer cf74fd686b9a4dbbb3937086a532fd9b. Status: Network error: Client connection negotiation failed: client connection to 127.14.127.252:43217: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 20:22:17.052384 21561 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:17.054270 21765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:17.054580 21561 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:17.054937 21561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 20:22:17.055038 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
uuid: "cf74fd686b9a4dbbb3937086a532fd9b"
format_stamp: "Formatted at 2026-09-23 20:22:16 on dist-test-slave-wmd0"
server_key: "a857d305188946b15cb9b2d1a73c7381"
server_key_iv: "47ef1608bcf30990228f66083fd8f60f"
server_key_version: "encryptionkey@0"
I20260923 20:22:17.058758 21561 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:17.063557 21561 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 20:22:17.063724 21561 log.cc:1196] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 20:22:17.064319 21561 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Deleting consensus metadata
I20260923 20:22:17.064579 21561 master_runner.cc:315] Copying system tablet from 127.14.127.254:46857
I20260923 20:22:17.067188 21561 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: tablet copy: Beginning tablet copy session from remote peer at address 127.14.127.254:46857
I20260923 20:22:17.080657 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.068093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.14.127.254:46857)
0923 20:22:17.068323 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:17.068349 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:17.068911 (+ 562us) server_negotiation.cc:407] Connection header received
0923 20:22:17.069109 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:17.069115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:17.069196 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:17.069306 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:17.070338 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.071401 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.072872 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.073399 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.073533 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.075431 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:17.075437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:17.075442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:17.075481 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:17.078104 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.078742 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.078750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.078754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.078895 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.079268 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.079275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.079278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.079550 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:17.079716 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:17.080074 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 20:22:17.080295 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:22:17.080749 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.068000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:44982)
0923 20:22:17.068786 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:17.068803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:17.068974 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:17.069325 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:17.069329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:17.069872 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:17.070187 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.070206 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.071533 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.071539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.072723 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.072734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.073571 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.073577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.073759 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.074421 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:17.074438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:17.075211 (+ 773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:17.078240 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:17.078246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:17.078250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:17.078604 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:17.079006 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:17.079011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:17.079015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:17.079161 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:17.079675 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:17.079681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:17.079841 (+ 160us) client_negotiation.cc:769] Sending connection context
0923 20:22:17.079985 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 20:22:17.080306 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":539,"threads_started":1}
I20260923 20:22:17.081696 21025 tablet_copy_service.cc:140] P bdd278a282f14dc69e4d6c04c9f016a0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cf74fd686b9a4dbbb3937086a532fd9b ({username='oryx', principal='oryx/127.14.127.252@KRBTEST.COM'} at 127.0.0.1:44982)
I20260923 20:22:17.081876 21025 tablet_copy_service.cc:161] P bdd278a282f14dc69e4d6c04c9f016a0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cf74fd686b9a4dbbb3937086a532fd9b at {username='oryx', principal='oryx/127.14.127.252@KRBTEST.COM'} at 127.0.0.1:44982: session id = cf74fd686b9a4dbbb3937086a532fd9b-00000000000000000000000000000000
I20260923 20:22:17.084338 21025 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 20:22:17.085934 21561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 20:22:17.090448 21561 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: tablet copy: Starting download of 0 data blocks...
I20260923 20:22:17.090751 21561 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: tablet copy: Starting download of 1 WAL segments...
I20260923 20:22:17.092880 21561 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 20:22:17.096043 21025 tablet_copy_service.cc:342] P bdd278a282f14dc69e4d6c04c9f016a0: Request end of tablet copy session cf74fd686b9a4dbbb3937086a532fd9b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.127.252@KRBTEST.COM'} at 127.0.0.1:44982
I20260923 20:22:17.096174 21025 tablet_copy_service.cc:433] P bdd278a282f14dc69e4d6c04c9f016a0: ending tablet copy session cf74fd686b9a4dbbb3937086a532fd9b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cf74fd686b9a4dbbb3937086a532fd9b
I20260923 20:22:17.107295 21561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:17.110222 21772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:17.115289 21773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:17.122290 21775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:17.128069 21561 server_base.cc:1034] running on GCE node
I20260923 20:22:17.128492 21561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:17.129235 21561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:17.130512 21561 hybrid_clock.cc:648] HybridClock initialized: now 1790194937130468 us; error 36 us; skew 500 ppm
I20260923 20:22:17.130877 21561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:17 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:17.133153 21561 init.cc:377] Logged in from keytab as oryx/127.14.127.252@KRBTEST.COM (short username oryx)
I20260923 20:22:17.134721 21561 webserver.cc:460] Webserver started at http://127.14.127.252:36701/ using document root <none> and password file <none>
I20260923 20:22:17.135138 21561 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:17.135288 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:17.138410 21561 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 20:22:17.140460 21782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:17.140939 21561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 20:22:17.141101 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
uuid: "cf74fd686b9a4dbbb3937086a532fd9b"
format_stamp: "Formatted at 2026-09-23 20:22:16 on dist-test-slave-wmd0"
server_key: "a857d305188946b15cb9b2d1a73c7381"
server_key_iv: "47ef1608bcf30990228f66083fd8f60f"
server_key_version: "encryptionkey@0"
I20260923 20:22:17.141364 21561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:17.183076 21561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:17.187404 21561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:17.213855 21561 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.252:43217
I20260923 20:22:17.213925 21833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.252:43217 every 8 connection(s)
I20260923 20:22:17.215078 21561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
I20260923 20:22:17.220738 21834 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:22:17.224897 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Bootstrap starting.
I20260923 20:22:17.241143 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:22:17.241824 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: Bootstrap complete.
I20260923 20:22:17.243661 21834 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } }
I20260923 20:22:17.244069 21834 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf74fd686b9a4dbbb3937086a532fd9b, State: Initialized, Role: LEARNER
I20260923 20:22:17.244403 21834 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } }
I20260923 20:22:17.245393 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:17.245849 21834 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:17.245913 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: LEARNER
I20260923 20:22:17.258261 21848 catalog_manager.cc:1259] Loaded cluster ID: 1f06babdafee4507aef3d034fc0b8ee1
I20260923 20:22:17.258383 21848 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: loading cluster ID for follower catalog manager: success
I20260923 20:22:17.262285 21848 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: acquiring CA information for follower catalog manager: success
I20260923 20:22:17.265039 21848 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:22:17.434525 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:17.434868 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
W20260923 20:22:17.438330 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:17.438748 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:17.438910 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:22:17.442451 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.434866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.0.0.1:43532)
0923 20:22:17.435138 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:17.435160 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:17.435298 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:17.436645 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:17.436650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:17.436681 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:17.436861 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.436871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.438269 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.438276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.440335 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.440351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.441497 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.441503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.441717 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.441757 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:22:17.441924 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 20:22:17.442087 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260923 20:22:17.442731 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.435087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.14.127.252:43217)
0923 20:22:17.436294 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:17.436299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:17.436317 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:22:17.436387 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:17.436391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:17.436486 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:17.436596 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:17.437027 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.438112 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.440484 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.441284 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.441527 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.441720 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:17.441942 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 20:22:17.442082 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":811,"threads_started":1}
I20260923 20:22:17.443746 21809 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260923 20:22:17.444362 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd278a282f14dc69e4d6c04c9f016a0. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: NON_VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: true } } }
I20260923 20:22:17.444761 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: LEARNER
I20260923 20:22:17.445029 21565 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: attempting to promote NON_VOTER cf74fd686b9a4dbbb3937086a532fd9b to VOTER
I20260923 20:22:17.446059 21565 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } }
I20260923 20:22:17.448611 21620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Refusing update from remote peer bdd278a282f14dc69e4d6c04c9f016a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 20:22:17.449502 21809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 LEARNER]: Refusing update from remote peer bdd278a282f14dc69e4d6c04c9f016a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 20:22:17.449353 21565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 20:22:17.450364 21565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 20:22:17.452435 21564 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cf74fd686b9a4dbbb3937086a532fd9b (127.14.127.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
W20260923 20:22:17.453455 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:17.453541 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:17.453655 21764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:17.455552 21620 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cf74fd686b9a4dbbb3937086a532fd9b (127.14.127.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:17.455991 21809 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cf74fd686b9a4dbbb3937086a532fd9b (127.14.127.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:17.457351 21566 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:17.457664 21566 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:17.457952 21566 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:17.458173 21566 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:17.458458 21566 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:17.458709 21566 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:17.459918 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:17.460222 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:22:17.461014 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:17.461355 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:22:17.466367 21764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab
--principal=oryx/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:36551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=42291
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901,127.14.127.252:43217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:17.467861 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:17.469262 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:17.485309 21855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:17.486034 21858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:17.486374 21856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:17.487773 21764 server_base.cc:1034] running on GCE node
I20260923 20:22:17.488610 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:17.489722 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:17.490991 21764 hybrid_clock.cc:648] HybridClock initialized: now 1790194937490888 us; error 52 us; skew 500 ppm
I20260923 20:22:17.491434 21764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:17 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:17.496227 21764 init.cc:377] Logged in from keytab as oryx/127.14.127.193@KRBTEST.COM (short username oryx)
I20260923 20:22:17.498590 21764 webserver.cc:460] Webserver started at http://127.14.127.193:42291/ using document root <none> and password file <none>
I20260923 20:22:17.499439 21764 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:17.499610 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:17.505694 21764 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 20:22:17.509054 21865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:17.510903 21764 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:22:17.511067 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "f71c2dde189346db80b51359c5319f6a"
format_stamp: "Formatted at 2026-09-23 20:22:14 on dist-test-slave-wmd0"
server_key: "3961ab0ded216b3b3d5d4ee1ca2913a4"
server_key_iv: "b1a82c1f5544046331c3703adb182e8f"
server_key_version: "encryptionkey@0"
I20260923 20:22:17.512877 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:17.529506 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:17.534646 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:17.534950 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:17.536056 21764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:17.537973 21764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:17.538077 21764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:17.538162 21764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:17.538234 21764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:17.576381 21764 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:36551
I20260923 20:22:17.576470 21977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:36551 every 8 connection(s)
I20260923 20:22:17.577822 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:17.583094 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 21764
I20260923 20:22:17.583460 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 21200
Sep 23 20:22:17 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
Sep 23 20:22:17 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.252@KRBTEST.COM
Sep 23 20:22:17 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.194@KRBTEST.COM
Sep 23 20:22:17 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.253@KRBTEST.COM
I20260923 20:22:17.606686 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.583672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:49795 (local address 127.14.127.252:43217)
0923 20:22:17.583889 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:17.583897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:17.584554 (+ 657us) server_negotiation.cc:407] Connection header received
0923 20:22:17.585812 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:17.585817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:17.585898 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:17.585990 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:17.588856 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.589972 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.592852 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.593515 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.593621 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.599397 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:17.599403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:17.599408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:17.599455 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:17.603176 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.603880 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.603885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.603889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.604024 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.604435 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.604440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.604443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.604855 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:17.604999 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:17.606050 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0923 20:22:17.606292 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:22:17.607606 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.580777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:48417 (local address 127.14.127.253:34901)
0923 20:22:17.582010 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:17.582016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:17.582717 (+ 701us) server_negotiation.cc:407] Connection header received
0923 20:22:17.585735 (+ 3018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:17.585741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:17.585815 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:17.585894 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:17.588877 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.589943 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.592879 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.593546 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.593652 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.600717 (+ 7065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:17.600729 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:17.600747 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:17.600803 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:17.604420 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.605340 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.605348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.605352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.605514 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.605969 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.605975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.605977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.606311 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:17.606441 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:17.607029 (+ 588us) server_negotiation.cc:299] Negotiation successful
0923 20:22:17.607247 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":889,"threads_started":1}
I20260923 20:22:17.607774 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.580759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:34007 (local address 127.14.127.254:46857)
0923 20:22:17.581009 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:17.581019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:17.582705 (+ 1686us) server_negotiation.cc:407] Connection header received
0923 20:22:17.585812 (+ 3107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:17.585817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:17.585908 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:17.585993 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:17.588875 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.589974 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.592865 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.593548 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.593651 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.599034 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:17.599042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:17.599048 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:17.599095 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:17.602704 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.605418 (+ 2714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.605425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.605430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.605579 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.606052 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.606057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.606062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.606399 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:17.606542 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:17.607128 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 20:22:17.607394 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:22:17.608544 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.584700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.14.127.194:42733)
0923 20:22:17.585385 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:17.585396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:17.585426 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:22:17.585505 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:17.585513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:17.585752 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:17.585916 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:17.588244 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.589923 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.591888 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.592735 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.592864 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.599938 (+ 7074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:17.599954 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:17.599971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:17.600013 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:17.604470 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.605365 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.605377 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.605384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.605560 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:17.606012 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:17.606021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:17.606027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:17.606434 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:17.606747 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:17.607931 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0923 20:22:17.608191 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":391,"threads_started":1}
I20260923 20:22:17.609843 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.581489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.14.127.193:48417)
0923 20:22:17.582585 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:17.582631 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:17.585424 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:17.586384 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:17.586398 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:17.587511 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:17.588656 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.588683 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.590139 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.590143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.592707 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.592721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.594039 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.594048 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.594418 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.595226 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:17.595266 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:17.600472 (+ 5206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:17.604552 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:17.604556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:17.604559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:17.605204 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:17.605666 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:17.605670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:17.605673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:17.605848 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:17.606520 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:17.606525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:17.606729 (+ 204us) client_negotiation.cc:769] Sending connection context
0923 20:22:17.607806 (+ 1077us) client_negotiation.cc:240] Negotiation successful
0923 20:22:17.608373 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":949,"spinlock_wait_cycles":88064,"thread_start_us":522,"threads_started":1}
I20260923 20:22:17.609938 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.581489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.193:34007)
0923 20:22:17.582521 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:17.582569 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:17.585433 (+ 2864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:17.586384 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:17.586395 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:17.587516 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:17.588664 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.588685 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.590143 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.590148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.592687 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.592702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.593995 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.594002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.594425 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.595542 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:17.595560 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:17.598741 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:17.602887 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:17.602897 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:17.604565 (+ 1668us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:17.605282 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:17.605729 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:17.605733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:17.605736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:17.605927 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:17.606653 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:17.606659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:17.606825 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 20:22:17.607975 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0923 20:22:17.608542 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":710,"spinlock_wait_cycles":8704,"thread_start_us":1466,"threads_started":1}
I20260923 20:22:17.609967 21323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:17.609895 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:17.583267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.14.127.193:49795)
0923 20:22:17.584422 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:17.584436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:17.585433 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:17.586389 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:17.586395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:17.587516 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:17.588664 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.588685 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.590145 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.590148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.592694 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:17.592703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:17.593995 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:17.594002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:17.594412 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:17.595499 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:17.595524 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:17.599177 (+ 3653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:17.603297 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:17.603303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:17.603315 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:17.603744 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:17.604130 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:17.604136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:17.604139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:17.604316 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:17.605081 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:17.605089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:17.605560 (+ 471us) client_negotiation.cc:769] Sending connection context
0923 20:22:17.607236 (+ 1676us) client_negotiation.cc:240] Negotiation successful
0923 20:22:17.608259 (+ 1023us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":593,"spinlock_wait_cycles":37248,"thread_start_us":881,"threads_started":1}
I20260923 20:22:17.611424 21979 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46857
I20260923 20:22:17.611603 21980 heartbeater.cc:347] Connected to a master server at 127.14.127.253:34901
I20260923 20:22:17.611747 21979 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:17.611773 21980 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:17.612692 21980 heartbeater.cc:511] Master 127.14.127.253:34901 requested a full tablet report, sending...
I20260923 20:22:17.614213 21979 heartbeater.cc:511] Master 127.14.127.254:46857 requested a full tablet report, sending...
I20260923 20:22:17.614694 21978 heartbeater.cc:347] Connected to a master server at 127.14.127.252:43217
I20260923 20:22:17.614812 21978 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:17.615190 21978 heartbeater.cc:511] Master 127.14.127.252:43217 requested a full tablet report, sending...
I20260923 20:22:17.615300 21610 ts_manager.cc:194] Registered new tserver with Master: f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551)
I20260923 20:22:17.615959 21005 ts_manager.cc:194] Re-registered known tserver with Master: f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551)
I20260923 20:22:17.616781 21005 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:34007
I20260923 20:22:17.618541 21799 ts_manager.cc:194] Registered new tserver with Master: f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551)
I20260923 20:22:17.699983 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 21200
I20260923 20:22:17.716632 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:42733
--local_ip_for_outbound_sockets=127.14.127.194
--tserver_master_addrs=127.14.127.254:46857
--webserver_port=46515
--webserver_interface=127.14.127.194
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab
--principal=oryx/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901,127.14.127.252:43217 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:18.056628 21989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:18.056960 21989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:18.057134 21989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:18.064374 21989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:18.064548 21989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:18.064736 21989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:18.075559 21989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab
--principal=oryx/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:42733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=46515
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901,127.14.127.252:43217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:18.076972 21989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:18.078321 21989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:18.091652 21994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:18.091735 21995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:18.091696 21997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:18.092558 21989 server_base.cc:1034] running on GCE node
I20260923 20:22:18.093189 21989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:18.094219 21989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:18.095459 21989 hybrid_clock.cc:648] HybridClock initialized: now 1790194938095405 us; error 52 us; skew 500 ppm
I20260923 20:22:18.095784 21989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:18.099078 21989 init.cc:377] Logged in from keytab as oryx/127.14.127.194@KRBTEST.COM (short username oryx)
I20260923 20:22:18.100616 21989 webserver.cc:460] Webserver started at http://127.14.127.194:46515/ using document root <none> and password file <none>
I20260923 20:22:18.101315 21989 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:18.101431 21989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:18.106817 21989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:22:18.109426 22004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:18.110801 21989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:18.110939 21989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "92efd8ec3f3a47e792d179a8b06cf105"
format_stamp: "Formatted at 2026-09-23 20:22:14 on dist-test-slave-wmd0"
server_key: "51f72e91ebffc89a5b7963a353d4ba85"
server_key_iv: "d46e93533aaa7a0d59671e4467c3c279"
server_key_version: "encryptionkey@0"
I20260923 20:22:18.111832 21989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:18.136528 21989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:18.141037 21989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:18.141300 21989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:18.142522 21989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:18.144057 21989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:18.144143 21989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:18.144222 21989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:18.144276 21989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:18.178805 21989 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:42733
I20260923 20:22:18.178916 22116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:42733 every 8 connection(s)
I20260923 20:22:18.180225 21989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:18.187207 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 21989
I20260923 20:22:18.187563 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 21333
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.195@KRBTEST.COM
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.253@KRBTEST.COM
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:18.224572 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.189195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.14.127.195:39771)
0923 20:22:18.199065 (+ 9870us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.199072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.199544 (+ 472us) server_negotiation.cc:407] Connection header received
0923 20:22:18.199772 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.199783 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.200108 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.200266 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.202819 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.204887 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.207451 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.208354 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.208478 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.215292 (+ 6814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.215306 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.215321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.215368 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.219259 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.220150 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.220159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.220165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.220327 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.221223 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.221230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.221233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.221651 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.221961 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.223814 (+ 1853us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.224123 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9722,"thread_start_us":9611,"threads_started":1}
I20260923 20:22:18.227140 21456 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.252@KRBTEST.COM
I20260923 20:22:18.239710 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.183533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:40951 (local address 127.14.127.253:34901)
0923 20:22:18.196695 (+ 13162us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.196700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.210425 (+ 13725us) server_negotiation.cc:407] Connection header received
0923 20:22:18.211064 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.211074 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.211197 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.211284 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.214458 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.215686 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.218605 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.219358 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.219463 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.224879 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.224886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.224892 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.224942 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.229216 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.235031 (+ 5815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.235041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.235048 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.235245 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.237557 (+ 2312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.237565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.237568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.237892 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.238055 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.239014 (+ 959us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.239280 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13038,"thread_start_us":12935,"threads_started":1}
I20260923 20:22:18.240983 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.184077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.194:45615)
0923 20:22:18.200414 (+ 16337us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.200470 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.209228 (+ 8758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.212064 (+ 2836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.212070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.213065 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:18.214240 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.214269 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.215809 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.215815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.218432 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.218447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.219768 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.219775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.220126 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.220969 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:18.221000 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:18.224327 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:18.233575 (+ 9248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.233579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.233583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.234850 (+ 1267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.236707 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.236715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.236721 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.236946 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.238179 (+ 1233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:18.238183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:18.238503 (+ 320us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.238864 (+ 361us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.239635 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15932,"mutex_wait_us":486,"spinlock_wait_cycles":45824,"thread_start_us":26947,"threads_started":1}
I20260923 20:22:18.241003 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.184077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.14.127.194:40951)
0923 20:22:18.210278 (+ 26201us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.210306 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.210825 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.211959 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.211972 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.213071 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:18.214244 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.214268 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.216012 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.216017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.218438 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.218454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.219782 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.219789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.220117 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.220949 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:18.220979 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:18.224152 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:18.233241 (+ 9089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.233250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.233267 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.234142 (+ 875us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.235551 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.235597 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.235604 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.237085 (+ 1481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.238035 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:18.238043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:18.238503 (+ 460us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.238864 (+ 361us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.239461 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25979,"mutex_wait_us":364,"spinlock_wait_cycles":110976,"thread_start_us":15655,"threads_started":1}
I20260923 20:22:18.242082 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.183654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:45615 (local address 127.14.127.254:46857)
0923 20:22:18.196970 (+ 13316us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.196974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.200578 (+ 3604us) server_negotiation.cc:407] Connection header received
0923 20:22:18.211629 (+ 11051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.211637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.211736 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.211864 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.214469 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.215599 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.218629 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.219321 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.219431 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.224598 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.224606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.224612 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.224657 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.233060 (+ 8403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.236271 (+ 3211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.236282 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.236288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.236518 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.237624 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.237634 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.237640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.238067 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.238231 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.240398 (+ 2167us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.240683 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13217,"thread_start_us":13302,"threads_started":1}
I20260923 20:22:18.242746 22119 heartbeater.cc:347] Connected to a master server at 127.14.127.253:34901
I20260923 20:22:18.243115 22119 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:18.243134 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.201490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:35889 (local address 127.14.127.252:43217)
0923 20:22:18.211960 (+ 10470us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.211968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.212385 (+ 417us) server_negotiation.cc:407] Connection header received
0923 20:22:18.212465 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.212472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.212568 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.212679 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.226022 (+ 13343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.227393 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.229421 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.230197 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.230320 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.234687 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.234697 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.234706 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.234755 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.239053 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.239991 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.239997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.240001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.240157 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.240673 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.240679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.240682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.241057 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.241230 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.242422 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.242708 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10337,"thread_start_us":10486,"threads_started":1}
I20260923 20:22:18.243700 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.201527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.14.127.194:35889)
0923 20:22:18.211529 (+ 10002us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.212096 (+ 567us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.212237 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.212880 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.212885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.225460 (+ 12575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:18.225853 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.225868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.227596 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.227602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.229242 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.229257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.230598 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.230605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.230842 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.231407 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:18.231430 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:18.234335 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:18.239216 (+ 4881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.239221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.239226 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.239837 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.240301 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.240306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.240310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.240527 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.241305 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:18.241311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:18.241497 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.241638 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.241957 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9786,"thread_start_us":8924,"threads_started":1}
I20260923 20:22:18.244045 22119 heartbeater.cc:511] Master 127.14.127.253:34901 requested a full tablet report, sending...
I20260923 20:22:18.244735 22118 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46857
I20260923 20:22:18.244877 22118 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:18.245241 22118 heartbeater.cc:511] Master 127.14.127.254:46857 requested a full tablet report, sending...
I20260923 20:22:18.246577 21005 ts_manager.cc:194] Re-registered known tserver with Master: 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194:42733)
I20260923 20:22:18.247165 21610 ts_manager.cc:194] Registered new tserver with Master: 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194:42733)
I20260923 20:22:18.248381 21005 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:45615
I20260923 20:22:18.248983 22117 heartbeater.cc:347] Connected to a master server at 127.14.127.252:43217
I20260923 20:22:18.249226 22117 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:18.250377 22117 heartbeater.cc:511] Master 127.14.127.252:43217 requested a full tablet report, sending...
I20260923 20:22:18.251695 21799 ts_manager.cc:194] Registered new tserver with Master: 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194:42733)
I20260923 20:22:18.328655 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 21333
I20260923 20:22:18.344702 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:39771
--local_ip_for_outbound_sockets=127.14.127.195
--tserver_master_addrs=127.14.127.254:46857
--webserver_port=34683
--webserver_interface=127.14.127.195
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab
--principal=oryx/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901,127.14.127.252:43217 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:22:18.620385 21979 heartbeater.cc:503] Master 127.14.127.254:46857 was elected leader, sending a full tablet report...
W20260923 20:22:18.673882 22130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:18.674216 22130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:18.674397 22130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:18.681725 22130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:18.681886 22130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:18.682070 22130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:18.692296 22130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab
--principal=oryx/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:39771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=34683
--tserver_master_addrs=127.14.127.254:46857,127.14.127.253:34901,127.14.127.252:43217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:18.693753 22130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:18.695122 22130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:18.709419 22136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:18.709491 22138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:18.710229 22135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:18.710182 22130 server_base.cc:1034] running on GCE node
I20260923 20:22:18.710983 22130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:18.711925 22130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:18.713132 22130 hybrid_clock.cc:648] HybridClock initialized: now 1790194938713093 us; error 30 us; skew 500 ppm
I20260923 20:22:18.713508 22130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:18.716962 22130 init.cc:377] Logged in from keytab as oryx/127.14.127.195@KRBTEST.COM (short username oryx)
I20260923 20:22:18.718739 22130 webserver.cc:460] Webserver started at http://127.14.127.195:34683/ using document root <none> and password file <none>
I20260923 20:22:18.719431 22130 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:18.719578 22130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:18.724947 22130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:22:18.727855 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:18.729188 22130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:18.729415 22130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "a27f9793e4a744d1a135d602dec9ee6e"
format_stamp: "Formatted at 2026-09-23 20:22:15 on dist-test-slave-wmd0"
server_key: "495369ff110cd9d3b326fd577f68acd0"
server_key_iv: "67f55f9cbc4f39cbece6a8c54e17afa1"
server_key_version: "encryptionkey@0"
I20260923 20:22:18.730417 22130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:18.746707 22130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:18.751075 22130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:18.751466 22130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:18.752516 22130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:18.754128 22130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:18.754236 22130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:18.754376 22130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:18.754456 22130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:18.790081 22130 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:39771
I20260923 20:22:18.790133 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:39771 every 8 connection(s)
I20260923 20:22:18.791623 22130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:18.795236 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 22130
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.253@KRBTEST.COM
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
Sep 23 20:22:18 dist-test-slave-wmd0 krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.252@KRBTEST.COM
I20260923 20:22:18.836826 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.797444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:54271 (local address 127.14.127.254:46857)
0923 20:22:18.811809 (+ 14365us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.811819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.811846 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:22:18.811943 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.811951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.812056 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.812205 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.815002 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.818968 (+ 3966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.820788 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.821676 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.821815 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.827167 (+ 5352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.827177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.827185 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.827236 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.832248 (+ 5012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.834406 (+ 2158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.834410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.834413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.834651 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.835097 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.835101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.835104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.835509 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.836112 (+ 603us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.836260 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.836484 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14193,"thread_start_us":14048,"threads_started":1}
I20260923 20:22:18.837989 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.796081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.195:54271)
0923 20:22:18.807102 (+ 11021us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.807131 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.811025 (+ 3894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.812549 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.812561 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.813677 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:18.814737 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.814762 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.819118 (+ 4356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.819123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.820663 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.820678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.822530 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.822536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.822750 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.823305 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:18.823326 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:18.826927 (+ 3601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:18.833659 (+ 6732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.833665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.833668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.834220 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.834793 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.834797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.834800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.834953 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.835643 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:18.835648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:18.835785 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.835919 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.836297 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10863,"mutex_wait_us":185,"thread_start_us":10656,"threads_started":1}
I20260923 20:22:18.839659 22259 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46857
I20260923 20:22:18.839987 22259 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:18.840931 22259 heartbeater.cc:511] Master 127.14.127.254:46857 requested a full tablet report, sending...
I20260923 20:22:18.842468 22268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.805892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:59687 (local address 127.14.127.253:34901)
0923 20:22:18.814128 (+ 8236us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.814135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.814153 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:22:18.814210 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.814214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.814283 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.814388 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.815770 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.816860 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.819812 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.820488 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.820611 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.827211 (+ 6600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.827218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.827223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.827288 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.832033 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.833041 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.833048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.833052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.833224 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.833938 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.833944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.833947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.834483 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.834637 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.838910 (+ 4273us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.842051 (+ 3141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8098,"thread_start_us":7962,"threads_started":1}
I20260923 20:22:18.843720 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.805717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.14.127.195:59687)
0923 20:22:18.809281 (+ 3564us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.809304 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.811013 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.814597 (+ 3584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.814601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.815204 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:18.815535 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.815547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.817057 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.817062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.819636 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.819655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.820952 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.820959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.821282 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.822142 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:18.822174 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:18.826993 (+ 4819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:18.832226 (+ 5233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.832235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.832254 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.832888 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.833425 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.833432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.833438 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.833762 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.834752 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:18.834762 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:18.835320 (+ 558us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.835643 (+ 323us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.843165 (+ 7522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3435,"mutex_wait_us":5,"thread_start_us":3306,"threads_started":1}
I20260923 20:22:18.845445 22260 heartbeater.cc:347] Connected to a master server at 127.14.127.253:34901
I20260923 20:22:18.845672 22260 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:18.847079 21005 ts_manager.cc:194] Re-registered known tserver with Master: a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195:39771)
I20260923 20:22:18.847506 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.794508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:43823 (local address 127.14.127.252:43217)
0923 20:22:18.813041 (+ 18533us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.813045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.813061 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:22:18.813109 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.813113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.813168 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.813337 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.815360 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.816629 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.821473 (+ 4844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.822174 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.822283 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.832483 (+ 10200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.832490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.832494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.832536 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.842239 (+ 9703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.843382 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.843389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.843393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.843541 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.844151 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.844159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.844164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.844607 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.844776 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.845988 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.846801 (+ 813us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18428,"thread_start_us":18317,"threads_started":1}
I20260923 20:22:18.848402 21005 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:54271
I20260923 20:22:18.849040 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.795172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.14.127.195:43823)
0923 20:22:18.805002 (+ 9830us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.805059 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.811013 (+ 5954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.813361 (+ 2348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.813374 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.813905 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:18.815121 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.815132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.816829 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.816834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.821288 (+ 4454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.821301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.823171 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.823179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.824811 (+ 1632us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.825524 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:18.825552 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:18.830272 (+ 4720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:18.842535 (+ 12263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.842540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.842543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.843033 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.843771 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:18.843776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:18.843779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:18.844008 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:18.844969 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:18.844974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:18.845142 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.845315 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.845730 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9481,"mutex_wait_us":3,"thread_start_us":9199,"threads_started":1}
I20260923 20:22:18.849479 22260 heartbeater.cc:511] Master 127.14.127.253:34901 requested a full tablet report, sending...
I20260923 20:22:18.850142 22258 heartbeater.cc:347] Connected to a master server at 127.14.127.252:43217
I20260923 20:22:18.850324 22258 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:18.850870 22258 heartbeater.cc:511] Master 127.14.127.252:43217 requested a full tablet report, sending...
I20260923 20:22:18.850987 21610 ts_manager.cc:194] Registered new tserver with Master: a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195:39771)
I20260923 20:22:18.852259 21799 ts_manager.cc:194] Registered new tserver with Master: a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195:39771)
I20260923 20:22:18.861661 22268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.816221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.14.127.253:34901)
0923 20:22:18.842741 (+ 26520us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.842747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.842763 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:22:18.842809 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.842814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.842932 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.843029 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.847506 (+ 4477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.849182 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.851239 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.851846 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.851959 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.855598 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.855603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.855606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.855636 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.858608 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.859452 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.859458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.859461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.859620 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.860068 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.860074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.860078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.860435 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.860694 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.861117 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.861280 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26410,"thread_start_us":36562,"threads_started":1}
I20260923 20:22:18.864914 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.824538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.14.127.254:46857)
0923 20:22:18.837095 (+ 12557us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.837101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.837118 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:22:18.837163 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.837167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.837240 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.837328 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.844236 (+ 6908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.846108 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.849904 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.850962 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.851102 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.857338 (+ 6236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.857344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.857349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.857390 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.860891 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.862745 (+ 1854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.862749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.862751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.862925 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.863307 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.863312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.863315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.863630 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.863886 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.864200 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.864536 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12464,"thread_start_us":16890,"threads_started":1}
I20260923 20:22:18.868443 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.837354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.14.127.252:43217)
0923 20:22:18.837841 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.837846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.843399 (+ 5553us) server_negotiation.cc:407] Connection header received
0923 20:22:18.847915 (+ 4516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.847920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.847998 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.848100 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:18.855257 (+ 7157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.856557 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.858436 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.859053 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.859217 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.862309 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:18.862315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:18.862320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:18.862357 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:18.865495 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.866179 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.866184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.866187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.866341 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:18.866808 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:18.866813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:18.866817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:18.867222 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:18.867519 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.867842 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.868069 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":239,"threads_started":1}
I20260923 20:22:18.869539 21005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:18.876711 21005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:18.923811 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.909872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36551 (local address 127.0.0.1:56220)
0923 20:22:18.910493 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.910519 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.910688 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.912546 (+ 1858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.912551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.912576 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:18.913057 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.913073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.916523 (+ 3450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.916528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.919460 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.919481 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.922691 (+ 3210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.922701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.922944 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.922988 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.923065 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.923209 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":1,"thread_start_us":285,"threads_started":1}
I20260923 20:22:18.924851 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.910367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:42733 (local address 127.0.0.1:57164)
0923 20:22:18.912497 (+ 2130us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.912516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.912655 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.913840 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.913848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.913897 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:18.914425 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.914441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.918159 (+ 3718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.918168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.921963 (+ 3795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.921982 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.923823 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.923830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.924183 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.924235 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.924336 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.924503 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1977,"spinlock_wait_cycles":1280,"thread_start_us":1811,"threads_started":1}
I20260923 20:22:18.925382 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.910792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.14.127.194:42733)
0923 20:22:18.913120 (+ 2328us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.913129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.913156 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:22:18.913269 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.913281 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.913582 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.915412 (+ 1830us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:18.915701 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.917950 (+ 2249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.922217 (+ 4267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.923557 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.923759 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.924090 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.924570 (+ 480us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.924743 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2171,"thread_start_us":2186,"threads_started":1}
I20260923 20:22:18.926455 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.910246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.14.127.193:36551)
0923 20:22:18.911641 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.911650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.911679 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:22:18.912003 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.912012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.912324 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.913307 (+ 983us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:18.913601 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.916334 (+ 2733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.920078 (+ 3744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.922377 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.925749 (+ 3372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.925958 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.926023 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.926177 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":1500,"threads_started":1}
I20260923 20:22:18.929658 21913 tablet_service.cc:1467] Processing CreateTablet for tablet 57e3bd91ae7f4484ab06ddc06f7254a2 (DEFAULT_TABLE table=test-table [id=50a53db92d074c37aab5053145e40cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:18.929490 22052 tablet_service.cc:1467] Processing CreateTablet for tablet 57e3bd91ae7f4484ab06ddc06f7254a2 (DEFAULT_TABLE table=test-table [id=50a53db92d074c37aab5053145e40cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:18.931037 21913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e3bd91ae7f4484ab06ddc06f7254a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:18.931265 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e3bd91ae7f4484ab06ddc06f7254a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:18.931353 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.912849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.14.127.195:39771)
0923 20:22:18.923238 (+ 10389us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:18.923246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:18.924233 (+ 987us) server_negotiation.cc:407] Connection header received
0923 20:22:18.924569 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:18.924581 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:18.924850 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:18.925052 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:18.925838 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.927868 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.929276 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.930291 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.930465 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.930656 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:18.930866 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 20:22:18.931030 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10195,"thread_start_us":10022,"threads_started":1}
I20260923 20:22:18.931857 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:18.912570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:39771 (local address 127.0.0.1:56798)
0923 20:22:18.924125 (+ 11555us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:18.924148 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:18.924268 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:18.925034 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:18.925038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:18.925064 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:18.925382 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.925401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.928015 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.928023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.929126 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:18.929138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:18.930462 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:18.930468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:18.930671 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:18.930717 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 20:22:18.930824 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:22:18.931615 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11433,"thread_start_us":11735,"threads_started":1}
I20260923 20:22:18.935211 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 57e3bd91ae7f4484ab06ddc06f7254a2 (DEFAULT_TABLE table=test-table [id=50a53db92d074c37aab5053145e40cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:18.936415 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e3bd91ae7f4484ab06ddc06f7254a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:18.949047 22295 tablet_bootstrap.cc:492] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Bootstrap starting.
I20260923 20:22:18.949975 22296 tablet_bootstrap.cc:492] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Bootstrap starting.
I20260923 20:22:18.950897 22297 tablet_bootstrap.cc:492] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Bootstrap starting.
I20260923 20:22:18.954205 22297 tablet_bootstrap.cc:654] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:18.954205 22295 tablet_bootstrap.cc:654] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:18.955525 22295 log.cc:824] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:18.955525 22297 log.cc:824] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:18.957785 22296 tablet_bootstrap.cc:654] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:18.958722 22295 tablet_bootstrap.cc:492] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: No bootstrap required, opened a new log
I20260923 20:22:18.958945 22297 tablet_bootstrap.cc:492] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: No bootstrap required, opened a new log
I20260923 20:22:18.959048 22295 ts_tablet_manager.cc:1397] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260923 20:22:18.959072 22296 log.cc:824] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:18.959211 22297 ts_tablet_manager.cc:1397] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260923 20:22:18.965826 22296 tablet_bootstrap.cc:492] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: No bootstrap required, opened a new log
I20260923 20:22:18.966109 22296 ts_tablet_manager.cc:1397] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260923 20:22:18.967111 22295 raft_consensus.cc:348] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:18.967595 22295 raft_consensus.cc:374] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:18.967694 22295 raft_consensus.cc:729] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71c2dde189346db80b51359c5319f6a, State: Initialized, Role: FOLLOWER
I20260923 20:22:18.968331 22295 consensus_queue.cc:260] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:18.970486 22297 raft_consensus.cc:348] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:18.971010 22297 raft_consensus.cc:374] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:18.971100 22297 raft_consensus.cc:729] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92efd8ec3f3a47e792d179a8b06cf105, State: Initialized, Role: FOLLOWER
I20260923 20:22:18.971688 22297 consensus_queue.cc:260] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:18.974164 22295 ts_tablet_manager.cc:1428] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260923 20:22:18.977171 22118 heartbeater.cc:503] Master 127.14.127.254:46857 was elected leader, sending a full tablet report...
I20260923 20:22:18.977896 22297 ts_tablet_manager.cc:1428] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s
I20260923 20:22:18.981345 22296 raft_consensus.cc:348] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:18.981819 22296 raft_consensus.cc:374] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:18.981928 22296 raft_consensus.cc:729] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27f9793e4a744d1a135d602dec9ee6e, State: Initialized, Role: FOLLOWER
I20260923 20:22:18.982558 22296 consensus_queue.cc:260] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:18.983814 22259 heartbeater.cc:503] Master 127.14.127.254:46857 was elected leader, sending a full tablet report...
I20260923 20:22:18.984409 22296 ts_tablet_manager.cc:1428] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
W20260923 20:22:19.058025 22261 tablet.cc:2369] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:22:19.081653 21981 tablet.cc:2369] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:22:19.153743 22302 raft_consensus.cc:482] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:19.154125 22302 raft_consensus.cc:504] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:19.156248 22302 leader_election.cc:288] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551), a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195:39771)
I20260923 20:22:19.164549 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.156936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:60265 (local address 127.14.127.195:39771)
0923 20:22:19.157265 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:19.157278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:19.158743 (+ 1465us) server_negotiation.cc:407] Connection header received
0923 20:22:19.158980 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:19.158984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:19.159040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:19.159134 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:19.160070 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.160762 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.162853 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.163629 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.163816 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.163891 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:19.164165 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 20:22:19.164297 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:22:19.164712 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.156629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:39771 (local address 127.14.127.194:60265)
0923 20:22:19.158583 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:19.158597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:19.158832 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:19.159150 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:19.159154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:19.159176 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:19.159899 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:19.159909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.160902 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.160908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:19.162706 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:19.162721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.163833 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.163839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:19.164009 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.164032 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:22:19.164139 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:22:19.164287 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1822,"mutex_wait_us":19,"thread_start_us":1523,"threads_started":1}
I20260923 20:22:19.164750 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.156984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:46459 (local address 127.14.127.193:36551)
0923 20:22:19.157579 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:19.157586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:19.158255 (+ 669us) server_negotiation.cc:407] Connection header received
0923 20:22:19.158532 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:19.158536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:19.158594 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:19.158748 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:19.159614 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.160497 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.162904 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.163680 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.163883 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.163957 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:19.164361 (+ 404us) server_negotiation.cc:299] Negotiation successful
0923 20:22:19.164491 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260923 20:22:19.165318 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.156469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36551 (local address 127.14.127.194:46459)
0923 20:22:19.158138 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:19.158154 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:19.158372 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:19.158769 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:19.158773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:19.158888 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:19.159436 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:19.159445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.160677 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.160681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:19.162759 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:19.162772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.163915 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.163921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:19.164076 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.164178 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 20:22:19.164333 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 20:22:19.164499 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"mutex_wait_us":1,"spinlock_wait_cycles":48512,"thread_start_us":1033,"threads_started":1}
I20260923 20:22:19.165865 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3bd91ae7f4484ab06ddc06f7254a2" candidate_uuid: "92efd8ec3f3a47e792d179a8b06cf105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27f9793e4a744d1a135d602dec9ee6e" is_pre_election: true
I20260923 20:22:19.166342 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3bd91ae7f4484ab06ddc06f7254a2" candidate_uuid: "92efd8ec3f3a47e792d179a8b06cf105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71c2dde189346db80b51359c5319f6a" is_pre_election: true
I20260923 20:22:19.166411 22213 raft_consensus.cc:2454] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92efd8ec3f3a47e792d179a8b06cf105 in term 0.
I20260923 20:22:19.166867 21933 raft_consensus.cc:2454] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92efd8ec3f3a47e792d179a8b06cf105 in term 0.
I20260923 20:22:19.167244 22005 leader_election.cc:302] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92efd8ec3f3a47e792d179a8b06cf105, a27f9793e4a744d1a135d602dec9ee6e; no voters:
I20260923 20:22:19.167745 22302 raft_consensus.cc:2781] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:19.167954 22302 raft_consensus.cc:482] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:19.168054 22302 raft_consensus.cc:3037] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:19.170236 22302 raft_consensus.cc:504] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:19.171031 22302 leader_election.cc:288] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [CANDIDATE]: Term 1 election: Requested vote from peers f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551), a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195:39771)
I20260923 20:22:19.171300 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3bd91ae7f4484ab06ddc06f7254a2" candidate_uuid: "92efd8ec3f3a47e792d179a8b06cf105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71c2dde189346db80b51359c5319f6a"
I20260923 20:22:19.171520 21933 raft_consensus.cc:3037] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:19.171532 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e3bd91ae7f4484ab06ddc06f7254a2" candidate_uuid: "92efd8ec3f3a47e792d179a8b06cf105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27f9793e4a744d1a135d602dec9ee6e"
I20260923 20:22:19.171727 22213 raft_consensus.cc:3037] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:19.174953 22213 raft_consensus.cc:2454] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92efd8ec3f3a47e792d179a8b06cf105 in term 1.
I20260923 20:22:19.174950 21933 raft_consensus.cc:2454] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92efd8ec3f3a47e792d179a8b06cf105 in term 1.
I20260923 20:22:19.175442 22008 leader_election.cc:302] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92efd8ec3f3a47e792d179a8b06cf105, f71c2dde189346db80b51359c5319f6a; no voters:
I20260923 20:22:19.175791 22302 raft_consensus.cc:2781] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:19.176244 22302 raft_consensus.cc:686] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 LEADER]: Becoming Leader. State: Replica: 92efd8ec3f3a47e792d179a8b06cf105, State: Running, Role: LEADER
I20260923 20:22:19.176865 22302 consensus_queue.cc:237] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } }
I20260923 20:22:19.182547 21005 catalog_manager.cc:5495] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194). New cstate: current_term: 1 leader_uuid: "92efd8ec3f3a47e792d179a8b06cf105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } health_report { overall_health: UNKNOWN } } }
W20260923 20:22:19.184070 22120 tablet.cc:2369] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:22:19.214710 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.209399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.14.127.194:42733)
0923 20:22:19.209579 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:19.209583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:19.209719 (+ 136us) server_negotiation.cc:407] Connection header received
0923 20:22:19.209906 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:19.209910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:19.209987 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:19.210111 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:19.210428 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.211198 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.212466 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.212871 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.212971 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.213450 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:19.214117 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:19.214260 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:19.214340 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 20:22:19.214417 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:22:19.223584 21933 raft_consensus.cc:1261] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 1 FOLLOWER]: Refusing update from remote peer 92efd8ec3f3a47e792d179a8b06cf105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:19.223584 22213 raft_consensus.cc:1261] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 1 FOLLOWER]: Refusing update from remote peer 92efd8ec3f3a47e792d179a8b06cf105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:19.224594 22306 consensus_queue.cc:1035] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:19.224978 22302 consensus_queue.cc:1035] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:19.239279 22311 mvcc.cc:205] Tried to move back new op lower bound from 7332638471049695232 to 7332638470872489984. Current Snapshot: MvccSnapshot[applied={T|T < 7332638471049695232}]
I20260923 20:22:19.255445 21005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:19.255666 21005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:19.263247 21005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdd278a282f14dc69e4d6c04c9f016a0: Sending DeleteTablet for 3 replicas of tablet 57e3bd91ae7f4484ab06ddc06f7254a2
I20260923 20:22:19.264497 21913 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e3bd91ae7f4484ab06ddc06f7254a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:19 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:56220
I20260923 20:22:19.264503 22052 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e3bd91ae7f4484ab06ddc06f7254a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:19 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57164
I20260923 20:22:19.265273 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e3bd91ae7f4484ab06ddc06f7254a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:19 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:56798
I20260923 20:22:19.265551 22320 tablet_replica.cc:330] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: stopping tablet replica
I20260923 20:22:19.266013 22320 raft_consensus.cc:2229] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:19.266536 22320 raft_consensus.cc:2258] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:19.267385 22319 tablet_replica.cc:330] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: stopping tablet replica
I20260923 20:22:19.267781 22319 raft_consensus.cc:2229] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:19.268407 22319 raft_consensus.cc:2258] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:19.269212 22320 ts_tablet_manager.cc:1905] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:19.270430 22319 ts_tablet_manager.cc:1905] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:19.277709 22320 ts_tablet_manager.cc:1918] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:19.277987 22319 ts_tablet_manager.cc:1918] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:19.278007 22320 log.cc:1196] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/57e3bd91ae7f4484ab06ddc06f7254a2
I20260923 20:22:19.278160 22319 log.cc:1196] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/57e3bd91ae7f4484ab06ddc06f7254a2
I20260923 20:22:19.278798 22319 ts_tablet_manager.cc:1939] T 57e3bd91ae7f4484ab06ddc06f7254a2 P 92efd8ec3f3a47e792d179a8b06cf105: Deleting consensus metadata
I20260923 20:22:19.278934 22320 ts_tablet_manager.cc:1939] T 57e3bd91ae7f4484ab06ddc06f7254a2 P f71c2dde189346db80b51359c5319f6a: Deleting consensus metadata
I20260923 20:22:19.280615 20989 catalog_manager.cc:4841] TS 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194:42733): tablet 57e3bd91ae7f4484ab06ddc06f7254a2 (table test-table [id=50a53db92d074c37aab5053145e40cfe]) successfully deleted
I20260923 20:22:19.280906 20992 catalog_manager.cc:4841] TS f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551): tablet 57e3bd91ae7f4484ab06ddc06f7254a2 (table test-table [id=50a53db92d074c37aab5053145e40cfe]) successfully deleted
I20260923 20:22:19.283936 22321 tablet_replica.cc:330] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: stopping tablet replica
I20260923 20:22:19.284416 22321 raft_consensus.cc:2229] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:19.284871 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 20973
I20260923 20:22:19.284937 22321 raft_consensus.cc:2258] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:19.285971 21035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:19.287192 22321 ts_tablet_manager.cc:1905] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:19.295641 22321 ts_tablet_manager.cc:1918] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:19.295980 22321 log.cc:1196] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/57e3bd91ae7f4484ab06ddc06f7254a2
I20260923 20:22:19.296832 22321 ts_tablet_manager.cc:1939] T 57e3bd91ae7f4484ab06ddc06f7254a2 P a27f9793e4a744d1a135d602dec9ee6e: Deleting consensus metadata
I20260923 20:22:19.306012 22268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.269830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.14.127.253:34901)
0923 20:22:19.270003 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:19.270009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:19.283485 (+ 13476us) server_negotiation.cc:407] Connection header received
0923 20:22:19.283563 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:19.283570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:19.283656 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:19.283775 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:19.286562 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.288143 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.290802 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.291946 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.292083 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.298514 (+ 6431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:19.298524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:19.298531 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:19.298581 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:19.302263 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:19.303596 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:19.303603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:19.303607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:19.303756 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:19.304279 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:19.304293 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:19.304296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:19.304672 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:19.304919 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:19.305442 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 20:22:19.305661 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:22:19.322713 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.283365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.14.127.252:43217)
0923 20:22:19.283551 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:19.283557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:19.306091 (+ 22534us) server_negotiation.cc:407] Connection header received
0923 20:22:19.306486 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:19.306493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:19.306578 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:19.306719 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:19.308861 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.309844 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.311714 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.312357 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.312492 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.315893 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:19.315901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:19.315908 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:19.315972 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:19.319798 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:19.320608 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:19.320613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:19.320617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:19.320751 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:19.321156 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:19.321161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:19.321165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:19.321499 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:19.321732 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:19.322091 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 20:22:19.322326 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:22:19.440047 20989 catalog_manager.cc:4841] TS a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195:39771): tablet 57e3bd91ae7f4484ab06ddc06f7254a2 (table test-table [id=50a53db92d074c37aab5053145e40cfe]) successfully deleted
I20260923 20:22:19.442166 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 20973
I20260923 20:22:19.486366 22268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.468250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.14.127.253:34901)
0923 20:22:19.468451 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:19.468457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:19.469348 (+ 891us) server_negotiation.cc:407] Connection header received
0923 20:22:19.469611 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:19.469618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:19.469696 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:19.469768 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:19.472149 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.473418 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.475405 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.476048 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.476139 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.479826 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:19.479832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:19.479837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:19.479879 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:19.483176 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:19.484004 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:19.484010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:19.484014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:19.484152 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:19.484607 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:19.484612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:19.484615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:19.484978 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:19.485244 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:19.485831 (+ 587us) server_negotiation.cc:299] Negotiation successful
0923 20:22:19.486044 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:22:19.496191 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:19.469532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.14.127.252:43217)
0923 20:22:19.469766 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:19.469775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:19.479009 (+ 9234us) server_negotiation.cc:407] Connection header received
0923 20:22:19.479340 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:19.479349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:19.479455 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:19.479541 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:19.482045 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.483923 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.486751 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:19.487412 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:19.487487 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:19.490976 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:19.490980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:19.490983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:19.491025 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:19.493526 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:19.494321 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:19.494324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:19.494326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:19.494440 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:19.494829 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:19.494832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:19.494834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:19.495098 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:19.495340 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:19.495682 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 20:22:19.495878 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:22:20.245007 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.244264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.194:45609)
0923 20:22:20.244648 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.244813 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":228,"threads_started":1}
W20260923 20:22:20.245410 22118 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:46857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
I20260923 20:22:20.250278 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.249419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.193:39963)
0923 20:22:20.249908 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.250075 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":234,"threads_started":1}
I20260923 20:22:20.250733 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.249819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.195:53053)
0923 20:22:20.250275 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.250429 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":398,"threads_started":1}
W20260923 20:22:20.251194 22259 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:46857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
W20260923 20:22:20.251418 21979 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:46857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
I20260923 20:22:20.762864 22334 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdd278a282f14dc69e4d6c04c9f016a0)
I20260923 20:22:20.763008 22335 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bdd278a282f14dc69e4d6c04c9f016a0)
I20260923 20:22:20.763221 22334 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } }
I20260923 20:22:20.763319 22335 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } }
I20260923 20:22:20.764726 22335 leader_election.cc:288] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdd278a282f14dc69e4d6c04c9f016a0 (127.14.127.254:46857), cf74fd686b9a4dbbb3937086a532fd9b (127.14.127.252:43217)
I20260923 20:22:20.764806 22334 leader_election.cc:288] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bdd278a282f14dc69e4d6c04c9f016a0 (127.14.127.254:46857), dff60975532f4b29868d003ed8fd3f92 (127.14.127.253:34901)
I20260923 20:22:20.778772 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.764866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35466)
0923 20:22:20.778127 (+ 13261us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.778367 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13095,"threads_started":1}
W20260923 20:22:20.779987 21597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
I20260923 20:22:20.781476 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.764855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35464)
0923 20:22:20.780957 (+ 16102us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.781171 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15962,"threads_started":1}
W20260923 20:22:20.782204 21786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
I20260923 20:22:20.788450 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.765242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:43217 (local address 127.0.0.1:38800)
0923 20:22:20.779019 (+ 13777us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.779041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:20.779447 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:20.780384 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:20.780450 (+ 66us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:20.780488 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:20.780749 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.780761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.782941 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.782949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.785313 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.785328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.787622 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.787631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.787836 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.787869 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:22:20.787970 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:22:20.788140 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13661,"thread_start_us":13926,"threads_started":1}
I20260923 20:22:20.788506 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.787673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35478)
0923 20:22:20.787907 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.788152 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:22:20.788760 21597 leader_election.cc:334] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdd278a282f14dc69e4d6c04c9f016a0 (127.14.127.254:46857): Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
I20260923 20:22:20.789281 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.765968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.14.127.252:43217)
0923 20:22:20.775173 (+ 9205us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:20.775179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:20.779145 (+ 3966us) server_negotiation.cc:407] Connection header received
0923 20:22:20.779663 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:20.779673 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:20.779776 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:20.779863 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:20.780957 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.782207 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.785500 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.787146 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.787399 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.787511 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:20.788706 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0923 20:22:20.788846 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":9089,"thread_start_us":13413,"threads_started":1}
I20260923 20:22:20.789960 21809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dff60975532f4b29868d003ed8fd3f92" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" is_pre_election: true
I20260923 20:22:20.790339 21809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff60975532f4b29868d003ed8fd3f92 in term 1.
I20260923 20:22:20.790993 21596 leader_election.cc:302] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf74fd686b9a4dbbb3937086a532fd9b, dff60975532f4b29868d003ed8fd3f92; no voters: bdd278a282f14dc69e4d6c04c9f016a0
I20260923 20:22:20.791427 22335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 20:22:20.791621 22335 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bdd278a282f14dc69e4d6c04c9f016a0)
I20260923 20:22:20.791782 22335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:22:20.792744 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.790912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35486)
0923 20:22:20.792414 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.792557 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1398}
I20260923 20:22:20.792763 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.765642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.14.127.253:34901)
0923 20:22:20.774779 (+ 9137us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:20.774785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:20.782328 (+ 7543us) server_negotiation.cc:407] Connection header received
0923 20:22:20.782394 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:20.782400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:20.782467 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:20.782560 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:20.784143 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.785406 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.790082 (+ 4676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.791433 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.791579 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.791668 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:20.792287 (+ 619us) server_negotiation.cc:299] Negotiation successful
0923 20:22:20.792423 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9022,"thread_start_us":8811,"threads_started":1}
W20260923 20:22:20.793184 21786 leader_election.cc:334] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bdd278a282f14dc69e4d6c04c9f016a0 (127.14.127.254:46857): Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
I20260923 20:22:20.793514 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.765294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:34901 (local address 127.0.0.1:50334)
0923 20:22:20.781655 (+ 16361us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.781673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:20.781781 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:20.783193 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:20.783198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:20.783230 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:20.783512 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.783523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.785615 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.785627 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.788904 (+ 3277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.788917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.791846 (+ 2929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.791854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.792068 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.792115 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 20:22:20.792943 (+ 828us) client_negotiation.cc:240] Negotiation successful
0923 20:22:20.793097 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16277,"mutex_wait_us":3,"thread_start_us":27607,"threads_started":1}
I20260923 20:22:20.794234 21620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dff60975532f4b29868d003ed8fd3f92" is_pre_election: true
I20260923 20:22:20.795229 22335 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } }
I20260923 20:22:20.795727 21620 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf74fd686b9a4dbbb3937086a532fd9b in current term 2: Already voted for candidate dff60975532f4b29868d003ed8fd3f92 in this term.
I20260923 20:22:20.796072 22335 leader_election.cc:288] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [CANDIDATE]: Term 2 election: Requested vote from peers bdd278a282f14dc69e4d6c04c9f016a0 (127.14.127.254:46857), cf74fd686b9a4dbbb3937086a532fd9b (127.14.127.252:43217)
I20260923 20:22:20.796414 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.796068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35502)
0923 20:22:20.796204 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.796292 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 20:22:20.796612 21809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dff60975532f4b29868d003ed8fd3f92" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cf74fd686b9a4dbbb3937086a532fd9b"
I20260923 20:22:20.796892 21809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:22:20.797008 21785 leader_election.cc:302] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf74fd686b9a4dbbb3937086a532fd9b; no voters: bdd278a282f14dc69e4d6c04c9f016a0, dff60975532f4b29868d003ed8fd3f92
I20260923 20:22:20.797643 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.797280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35506)
0923 20:22:20.797431 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.797506 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:22:20.797873 21597 leader_election.cc:334] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bdd278a282f14dc69e4d6c04c9f016a0 (127.14.127.254:46857): Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
I20260923 20:22:20.799360 21809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff60975532f4b29868d003ed8fd3f92 in term 2.
I20260923 20:22:20.799531 22334 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:22:20.799819 21596 leader_election.cc:302] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf74fd686b9a4dbbb3937086a532fd9b, dff60975532f4b29868d003ed8fd3f92; no voters: bdd278a282f14dc69e4d6c04c9f016a0
I20260923 20:22:20.800127 22335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:22:20.800590 22335 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [term 2 LEADER]: Becoming Leader. State: Replica: dff60975532f4b29868d003ed8fd3f92, State: Running, Role: LEADER
I20260923 20:22:20.800963 22335 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } }
I20260923 20:22:20.802590 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff60975532f4b29868d003ed8fd3f92. Latest consensus state: current_term: 2 leader_uuid: "dff60975532f4b29868d003ed8fd3f92" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:20.803103 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:20.803953 22346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:20.809194 22346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:20.810175 22346 catalog_manager.cc:1259] Loaded cluster ID: 1f06babdafee4507aef3d034fc0b8ee1
I20260923 20:22:20.810245 22346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:20.811786 22346 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:20.812594 22346 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Loaded TSK: 0
I20260923 20:22:20.813409 22346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:22:20.837317 21610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:20.844395 21610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:20.848515 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.848036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35524)
0923 20:22:20.848239 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.848335 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:22:20.848623 21809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [term 2 FOLLOWER]: Refusing update from remote peer dff60975532f4b29868d003ed8fd3f92: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 20:22:20.849455 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.849096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35530)
0923 20:22:20.849239 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.849330 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 20:22:20.849442 22344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260923 20:22:20.849802 21597 consensus_peers.cc:487] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 -> Peer bdd278a282f14dc69e4d6c04c9f016a0 (127.14.127.254:46857): Couldn't send request to peer bdd278a282f14dc69e4d6c04c9f016a0. Status: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 20:22:20.852352 22334 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff60975532f4b29868d003ed8fd3f92. Latest consensus state: current_term: 2 leader_uuid: "dff60975532f4b29868d003ed8fd3f92" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:20.852602 22334 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:22:20.854076 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "dff60975532f4b29868d003ed8fd3f92" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:20.854348 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:20.854707 22334 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "dff60975532f4b29868d003ed8fd3f92" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bdd278a282f14dc69e4d6c04c9f016a0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46857 } } peers { permanent_uuid: "dff60975532f4b29868d003ed8fd3f92" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 34901 } attrs { promote: false } } peers { permanent_uuid: "cf74fd686b9a4dbbb3937086a532fd9b" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 43217 } attrs { promote: false } } }
I20260923 20:22:20.854972 22334 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74fd686b9a4dbbb3937086a532fd9b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:22:20.877992 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.862407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.14.127.193:36551)
0923 20:22:20.871829 (+ 9422us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:20.871834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:20.871858 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:22:20.871940 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:20.871947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:20.872020 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:20.872130 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:20.872559 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.873623 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.875898 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.877058 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.877230 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.877311 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:20.877569 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 20:22:20.877694 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9271,"thread_start_us":9125,"threads_started":1}
I20260923 20:22:20.878005 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.862278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36551 (local address 127.0.0.1:41736)
0923 20:22:20.862493 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.862509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:20.862706 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:20.872148 (+ 9442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:20.872152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:20.872175 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:20.872363 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.872378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.873783 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.873792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.875751 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.875760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.877231 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.877237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.877406 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.877435 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 20:22:20.877545 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:22:20.877680 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":22}
I20260923 20:22:20.878254 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.862843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:39771 (local address 127.0.0.1:55742)
0923 20:22:20.862968 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.862988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:20.863081 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:20.871268 (+ 8187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:20.871273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:20.871307 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:20.871633 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.871646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.873207 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.873212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.875469 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.875487 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.877493 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.877498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.877672 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.877707 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:22:20.877788 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 20:22:20.877914 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":61}
I20260923 20:22:20.878939 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.863229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.14.127.195:39771)
0923 20:22:20.870937 (+ 7708us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:20.870941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:20.870978 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:22:20.871055 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:20.871060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:20.871127 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:20.871241 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:20.871837 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.873038 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.875656 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.877077 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.877241 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.877339 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:20.878466 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0923 20:22:20.878649 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7594,"thread_start_us":7491,"threads_started":1}
I20260923 20:22:20.879360 21913 tablet_service.cc:1467] Processing CreateTablet for tablet 8574e7463f8a4e4f9eb7af75fe654d4e (DEFAULT_TABLE table=test-table [id=4a71ca8340854092b09756711ffe9041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:20.879690 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.863358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:42733 (local address 127.0.0.1:40880)
0923 20:22:20.872728 (+ 9370us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.872752 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:20.872945 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:20.873438 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:20.873442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:20.873463 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:20.873974 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.873985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.875059 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.875064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.877147 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:20.877163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.878963 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.878969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:20.879134 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.879164 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:22:20.879257 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 20:22:20.879396 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9237,"thread_start_us":9043,"threads_started":1}
I20260923 20:22:20.879735 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.863471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.14.127.194:42733)
0923 20:22:20.863940 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:20.863948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:20.872847 (+ 8899us) server_negotiation.cc:407] Connection header received
0923 20:22:20.873097 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:20.873104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:20.873179 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:20.873242 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:20.874142 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.874912 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.877315 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:20.878494 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:20.878610 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:20.878696 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:20.879290 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 20:22:20.879444 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":213,"threads_started":1}
I20260923 20:22:20.879859 21913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8574e7463f8a4e4f9eb7af75fe654d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:20.880280 22193 tablet_service.cc:1467] Processing CreateTablet for tablet 8574e7463f8a4e4f9eb7af75fe654d4e (DEFAULT_TABLE table=test-table [id=4a71ca8340854092b09756711ffe9041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:20.880486 22052 tablet_service.cc:1467] Processing CreateTablet for tablet 8574e7463f8a4e4f9eb7af75fe654d4e (DEFAULT_TABLE table=test-table [id=4a71ca8340854092b09756711ffe9041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:20.880703 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8574e7463f8a4e4f9eb7af75fe654d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:20.880879 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8574e7463f8a4e4f9eb7af75fe654d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:20.884948 22357 tablet_bootstrap.cc:492] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: Bootstrap starting.
I20260923 20:22:20.886734 22357 tablet_bootstrap.cc:654] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:20.887435 22358 tablet_bootstrap.cc:492] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: Bootstrap starting.
I20260923 20:22:20.889694 22358 tablet_bootstrap.cc:654] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:20.891322 22359 tablet_bootstrap.cc:492] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: Bootstrap starting.
I20260923 20:22:20.892669 22359 tablet_bootstrap.cc:654] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:20.894757 22357 tablet_bootstrap.cc:492] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: No bootstrap required, opened a new log
I20260923 20:22:20.894920 22357 ts_tablet_manager.cc:1397] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260923 20:22:20.894954 22358 tablet_bootstrap.cc:492] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: No bootstrap required, opened a new log
I20260923 20:22:20.895104 22358 ts_tablet_manager.cc:1397] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 20:22:20.895113 22359 tablet_bootstrap.cc:492] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: No bootstrap required, opened a new log
I20260923 20:22:20.895264 22359 ts_tablet_manager.cc:1397] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 20:22:20.895769 22357 raft_consensus.cc:348] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:20.895888 22358 raft_consensus.cc:348] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:20.896063 22357 raft_consensus.cc:374] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:20.896135 22358 raft_consensus.cc:374] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:20.896157 22357 raft_consensus.cc:729] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71c2dde189346db80b51359c5319f6a, State: Initialized, Role: FOLLOWER
I20260923 20:22:20.896024 22359 raft_consensus.cc:348] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:20.896231 22358 raft_consensus.cc:729] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27f9793e4a744d1a135d602dec9ee6e, State: Initialized, Role: FOLLOWER
I20260923 20:22:20.896265 22359 raft_consensus.cc:374] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:20.896423 22359 raft_consensus.cc:729] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92efd8ec3f3a47e792d179a8b06cf105, State: Initialized, Role: FOLLOWER
I20260923 20:22:20.896435 22357 consensus_queue.cc:260] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:20.896466 22358 consensus_queue.cc:260] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:20.896672 22359 consensus_queue.cc:260] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:20.899339 22359 ts_tablet_manager.cc:1428] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260923 20:22:20.908450 22119 heartbeater.cc:503] Master 127.14.127.253:34901 was elected leader, sending a full tablet report...
I20260923 20:22:20.911504 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.899978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.194:52079)
0923 20:22:20.910569 (+ 10591us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.911126 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10415,"thread_start_us":10882,"threads_started":1}
I20260923 20:22:20.913131 22357 ts_tablet_manager.cc:1428] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260923 20:22:20.913399 22358 ts_tablet_manager.cc:1428] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
I20260923 20:22:20.913722 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.912261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.194:58565)
0923 20:22:20.913481 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.913573 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1090,"threads_started":1}
I20260923 20:22:20.915868 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.914334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.193:48329)
0923 20:22:20.915370 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.915536 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":883,"threads_started":1}
I20260923 20:22:20.917086 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.916619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.193:56147)
0923 20:22:20.916854 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.916936 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260923 20:22:20.917495 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.914531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.195:47313)
0923 20:22:20.917079 (+ 2548us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.917222 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2366,"threads_started":1}
I20260923 20:22:20.917794 22260 heartbeater.cc:503] Master 127.14.127.253:34901 was elected leader, sending a full tablet report...
I20260923 20:22:20.918740 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:20.918224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.195:42797)
0923 20:22:20.918498 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:20.918577 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260923 20:22:20.919078 21980 heartbeater.cc:503] Master 127.14.127.253:34901 was elected leader, sending a full tablet report...
I20260923 20:22:21.033252 22364 raft_consensus.cc:482] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:21.033463 22364 raft_consensus.cc:504] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:21.035043 22364 leader_election.cc:288] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551), 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194:42733)
I20260923 20:22:21.040630 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.035598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:58915 (local address 127.14.127.194:42733)
0923 20:22:21.035791 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:21.035797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:21.035823 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:22:21.035884 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:21.035887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:21.035947 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:21.036071 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:21.036595 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.037258 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.038980 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.039725 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.039903 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:21.039987 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:21.040248 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 20:22:21.040376 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:22:21.040666 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.035300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:42733 (local address 127.14.127.195:58915)
0923 20:22:21.035443 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.035465 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:21.035593 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:21.036108 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:21.036113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:21.036167 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:21.036438 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:21.036446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.037396 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.037402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:21.038824 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:21.038837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.039919 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.039926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:21.040084 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:21.040108 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:22:21.040212 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:22:21.040367 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"spinlock_wait_cycles":5504}
I20260923 20:22:21.041092 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.035534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:42103 (local address 127.14.127.193:36551)
0923 20:22:21.035757 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:21.035763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:21.036223 (+ 460us) server_negotiation.cc:407] Connection header received
0923 20:22:21.036431 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:21.036440 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:21.036501 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:21.036602 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:21.037090 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.037788 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.039466 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.040178 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.040353 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:21.040436 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:21.040723 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 20:22:21.040847 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:22:21.041190 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.035300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36551 (local address 127.14.127.195:42103)
0923 20:22:21.036116 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.036129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:21.036278 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:21.036615 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:21.036620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:21.036641 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:21.036935 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:21.036943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.037930 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.037936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:21.039300 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:21.039311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.040362 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.040368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:21.040549 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:21.040571 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:22:21.040685 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 20:22:21.040801 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":169,"thread_start_us":380,"threads_started":1}
I20260923 20:22:21.041735 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574e7463f8a4e4f9eb7af75fe654d4e" candidate_uuid: "a27f9793e4a744d1a135d602dec9ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92efd8ec3f3a47e792d179a8b06cf105" is_pre_election: true
I20260923 20:22:21.041930 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574e7463f8a4e4f9eb7af75fe654d4e" candidate_uuid: "a27f9793e4a744d1a135d602dec9ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71c2dde189346db80b51359c5319f6a" is_pre_election: true
I20260923 20:22:21.042166 21933 raft_consensus.cc:2454] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a27f9793e4a744d1a135d602dec9ee6e in term 0.
I20260923 20:22:21.042205 22072 raft_consensus.cc:2454] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a27f9793e4a744d1a135d602dec9ee6e in term 0.
I20260923 20:22:21.042814 22149 leader_election.cc:302] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 92efd8ec3f3a47e792d179a8b06cf105, a27f9793e4a744d1a135d602dec9ee6e, f71c2dde189346db80b51359c5319f6a; no voters:
I20260923 20:22:21.043251 22364 raft_consensus.cc:2781] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:21.043380 22364 raft_consensus.cc:482] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:21.043471 22364 raft_consensus.cc:3037] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:21.045601 22364 raft_consensus.cc:504] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:21.046217 22364 leader_election.cc:288] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [CANDIDATE]: Term 1 election: Requested vote from peers f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551), 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194:42733)
I20260923 20:22:21.046707 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574e7463f8a4e4f9eb7af75fe654d4e" candidate_uuid: "a27f9793e4a744d1a135d602dec9ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92efd8ec3f3a47e792d179a8b06cf105"
I20260923 20:22:21.046705 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8574e7463f8a4e4f9eb7af75fe654d4e" candidate_uuid: "a27f9793e4a744d1a135d602dec9ee6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71c2dde189346db80b51359c5319f6a"
I20260923 20:22:21.046905 21933 raft_consensus.cc:3037] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:21.046905 22072 raft_consensus.cc:3037] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:21.049876 21933 raft_consensus.cc:2454] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a27f9793e4a744d1a135d602dec9ee6e in term 1.
I20260923 20:22:21.049876 22072 raft_consensus.cc:2454] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a27f9793e4a744d1a135d602dec9ee6e in term 1.
I20260923 20:22:21.050262 22146 leader_election.cc:302] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92efd8ec3f3a47e792d179a8b06cf105, a27f9793e4a744d1a135d602dec9ee6e; no voters:
I20260923 20:22:21.050602 22364 raft_consensus.cc:2781] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:21.051113 22364 raft_consensus.cc:686] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 1 LEADER]: Becoming Leader. State: Replica: a27f9793e4a744d1a135d602dec9ee6e, State: Running, Role: LEADER
I20260923 20:22:21.052079 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.051713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.195:43091)
0923 20:22:21.051852 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.051964 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 20:22:21.051951 22364 consensus_queue.cc:237] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } }
I20260923 20:22:21.055317 21609 catalog_manager.cc:5495] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e reported cstate change: term changed from 0 to 1, leader changed from <none> to a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195). New cstate: current_term: 1 leader_uuid: "a27f9793e4a744d1a135d602dec9ee6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a27f9793e4a744d1a135d602dec9ee6e" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 } health_report { overall_health: UNKNOWN } } }
I20260923 20:22:21.075742 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.070749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.14.127.195:39771)
0923 20:22:21.070991 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:21.070996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:21.071011 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:22:21.071098 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:21.071101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:21.071172 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:21.071368 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:21.071663 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.072447 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.073630 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.074006 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.074104 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:21.074460 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:21.075141 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:21.075255 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:21.075384 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 20:22:21.075451 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:22:21.079882 21933 raft_consensus.cc:1261] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 1 FOLLOWER]: Refusing update from remote peer a27f9793e4a744d1a135d602dec9ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:21.080219 22072 raft_consensus.cc:1261] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 FOLLOWER]: Refusing update from remote peer a27f9793e4a744d1a135d602dec9ee6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:21.080583 22364 consensus_queue.cc:1035] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71c2dde189346db80b51359c5319f6a" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:21.080965 22371 consensus_queue.cc:1035] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92efd8ec3f3a47e792d179a8b06cf105" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:21.083684 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.083123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.194:56153)
0923 20:22:21.083372 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.083485 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260923 20:22:21.084079 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.083477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.193:43953)
0923 20:22:21.083708 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.083817 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260923 20:22:21.090026 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.089006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.195:51059)
0923 20:22:21.089643 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.089773 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260923 20:22:21.091572 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.091106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.194:50177)
0923 20:22:21.091323 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.091426 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 20:22:21.097891 22376 mvcc.cc:205] Tried to move back new op lower bound from 7332638478655549440 to 7332638478552768512. Current Snapshot: MvccSnapshot[applied={T|T < 7332638478655549440 or (T in {7332638478655549440})}]
I20260923 20:22:21.102525 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.102017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.195:48075)
0923 20:22:21.102220 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.102329 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:22:21.103988 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.103520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.14.127.193:47309)
0923 20:22:21.103731 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.103827 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 20:22:21.114705 21609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:21.115010 21609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:21.120704 21609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dff60975532f4b29868d003ed8fd3f92: Sending DeleteTablet for 3 replicas of tablet 8574e7463f8a4e4f9eb7af75fe654d4e
I20260923 20:22:21.121757 21913 tablet_service.cc:1514] Processing DeleteTablet for tablet 8574e7463f8a4e4f9eb7af75fe654d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:21 UTC) from {username='oryx', principal='oryx/127.14.127.253@KRBTEST.COM'} at 127.0.0.1:41736
I20260923 20:22:21.122324 22052 tablet_service.cc:1514] Processing DeleteTablet for tablet 8574e7463f8a4e4f9eb7af75fe654d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:21 UTC) from {username='oryx', principal='oryx/127.14.127.253@KRBTEST.COM'} at 127.0.0.1:40880
I20260923 20:22:21.122522 22193 tablet_service.cc:1514] Processing DeleteTablet for tablet 8574e7463f8a4e4f9eb7af75fe654d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:21 UTC) from {username='oryx', principal='oryx/127.14.127.253@KRBTEST.COM'} at 127.0.0.1:55742
I20260923 20:22:21.123212 22386 tablet_replica.cc:330] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: stopping tablet replica
I20260923 20:22:21.123413 22386 raft_consensus.cc:2229] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:21.124004 22386 raft_consensus.cc:2258] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:21.124444 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 21764
I20260923 20:22:21.125161 22386 ts_tablet_manager.cc:1905] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:21.127211 22385 tablet_replica.cc:330] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: stopping tablet replica
I20260923 20:22:21.127493 22385 raft_consensus.cc:2229] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:21.127864 22385 raft_consensus.cc:2258] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:21.129104 22385 ts_tablet_manager.cc:1905] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:21.131870 22386 ts_tablet_manager.cc:1918] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:21.132014 22386 log.cc:1196] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/8574e7463f8a4e4f9eb7af75fe654d4e
I20260923 20:22:21.132501 22386 ts_tablet_manager.cc:1939] T 8574e7463f8a4e4f9eb7af75fe654d4e P a27f9793e4a744d1a135d602dec9ee6e: Deleting consensus metadata
I20260923 20:22:21.133701 21594 catalog_manager.cc:4841] TS a27f9793e4a744d1a135d602dec9ee6e (127.14.127.195:39771): tablet 8574e7463f8a4e4f9eb7af75fe654d4e (table test-table [id=4a71ca8340854092b09756711ffe9041]) successfully deleted
I20260923 20:22:21.135587 22385 ts_tablet_manager.cc:1918] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:21.135743 22385 log.cc:1196] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/8574e7463f8a4e4f9eb7af75fe654d4e
I20260923 20:22:21.136173 22385 ts_tablet_manager.cc:1939] T 8574e7463f8a4e4f9eb7af75fe654d4e P 92efd8ec3f3a47e792d179a8b06cf105: Deleting consensus metadata
I20260923 20:22:21.137113 21594 catalog_manager.cc:4841] TS 92efd8ec3f3a47e792d179a8b06cf105 (127.14.127.194:42733): tablet 8574e7463f8a4e4f9eb7af75fe654d4e (table test-table [id=4a71ca8340854092b09756711ffe9041]) successfully deleted
I20260923 20:22:21.143299 22384 tablet_replica.cc:330] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: stopping tablet replica
I20260923 20:22:21.143513 22384 raft_consensus.cc:2229] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:21.143826 22384 raft_consensus.cc:2258] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:21.144767 22384 ts_tablet_manager.cc:1905] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:21.149817 22384 ts_tablet_manager.cc:1918] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:21.149964 22384 log.cc:1196] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/8574e7463f8a4e4f9eb7af75fe654d4e
I20260923 20:22:21.150424 22384 ts_tablet_manager.cc:1939] T 8574e7463f8a4e4f9eb7af75fe654d4e P f71c2dde189346db80b51359c5319f6a: Deleting consensus metadata
I20260923 20:22:21.151769 21597 catalog_manager.cc:4841] TS f71c2dde189346db80b51359c5319f6a (127.14.127.193:36551): tablet 8574e7463f8a4e4f9eb7af75fe654d4e (table test-table [id=4a71ca8340854092b09756711ffe9041]) successfully deleted
I20260923 20:22:21.156013 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.125681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.14.127.193:36551)
0923 20:22:21.125901 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:21.125909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:21.140307 (+ 14398us) server_negotiation.cc:407] Connection header received
0923 20:22:21.140569 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:21.140573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:21.140647 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:21.140783 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:21.142277 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.143620 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.145367 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.146017 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.146202 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:21.149635 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:21.149656 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:21.149676 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:21.149726 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:21.153175 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:21.153984 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:21.153991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:21.153993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:21.154112 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:21.154502 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:21.154507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:21.154510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:21.154934 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:21.155166 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:21.155529 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 20:22:21.155708 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:22:21.156625 21973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:21.271925 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 21764
I20260923 20:22:21.289486 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 21989
I20260923 20:22:21.306054 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.290337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.14.127.194:42733)
0923 20:22:21.290509 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:21.290513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:21.290580 (+ 67us) server_negotiation.cc:407] Connection header received
0923 20:22:21.290839 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:21.290861 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:21.290979 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:21.291068 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:21.292762 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.294091 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.295711 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.296363 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.296495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:21.299076 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:21.299095 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:21.299116 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:21.299162 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:21.303061 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:21.303806 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:21.303815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:21.303819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:21.303970 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:21.304337 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:21.304343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:21.304348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:21.304818 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:21.305115 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:21.305447 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 20:22:21.305666 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:22:21.306835 22112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:21.428630 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 21989
I20260923 20:22:21.451756 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 22130
I20260923 20:22:21.467706 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.452580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.14.127.195:39771)
0923 20:22:21.452775 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:21.452781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:21.452803 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:22:21.452976 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:21.452987 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:21.453094 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:21.453222 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:21.454569 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.455821 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.457034 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:21.457707 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:21.457863 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:21.460388 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:21.460406 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:21.460436 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:21.460486 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:21.464581 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:21.465294 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:21.465304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:21.465307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:21.465512 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:21.466016 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:21.466025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:21.466032 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:21.466486 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:21.466882 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:21.467047 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 20:22:21.467290 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:22:21.468536 22253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:21.470232 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.469331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35532)
0923 20:22:21.469904 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.470006 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":342,"threads_started":1}
I20260923 20:22:21.472082 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:21.471603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46857 (local address 127.0.0.1:35534)
0923 20:22:21.471751 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:21.471876 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:46857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:22:21.597013 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 22130
I20260923 20:22:21.617702 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 21466
I20260923 20:22:21.618788 21640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:21.741333 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 21466
W20260923 20:22:21.762982 21784 connection.cc:535] server connection from 127.0.0.1:38800 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 20:22:21.763341 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 21561
I20260923 20:22:21.764407 21829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:21.881804 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 21561
2026-09-23T20:22:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9702 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:22:21 dist-test-slave-wmd0 krb5kdc[22397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:22:21 dist-test-slave-wmd0 krb5kdc[22397](info): set up 2 sockets
Sep 23 20:22:21 dist-test-slave-wmd0 krb5kdc[22397](info): commencing operation
krb5kdc: starting...
W20260923 20:22:22.987653 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:22:23 dist-test-slave-wmd0 krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.254.keytab.
Entry for principal oryx/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.254.keytab.
I20260923 20:22:23.194403 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46083
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.254.keytab
--principal=oryx/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:23.518646 22410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:23.518934 22410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:23.519116 22410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:23.526435 22410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:23.526543 22410 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:22:23.526613 22410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:23.526741 22410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:23.526836 22410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:23.526928 22410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:23.537016 22410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:46083
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.254.keytab
--principal=oryx/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:23.538312 22410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:23.539677 22410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:23.551705 22418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:23.551776 22416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:23.551669 22415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:23.552670 22410 server_base.cc:1034] running on GCE node
I20260923 20:22:23.553511 22410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:23.554970 22410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:23.556233 22410 hybrid_clock.cc:648] HybridClock initialized: now 1790194943556163 us; error 47 us; skew 500 ppm
I20260923 20:22:23.556603 22410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:23 dist-test-slave-wmd0 krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194943, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:23.559763 22410 init.cc:377] Logged in from keytab as oryx/127.14.127.254@KRBTEST.COM (short username oryx)
I20260923 20:22:23.561187 22410 webserver.cc:460] Webserver started at http://127.14.127.254:40563/ using document root <none> and password file <none>
I20260923 20:22:23.561842 22410 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:23.561930 22410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:23.562194 22410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:23.565163 22410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "32e2d393000c403880a2519dcc9ffdc0"
format_stamp: "Formatted at 2026-09-23 20:22:23 on dist-test-slave-wmd0"
server_key: "1a00372ab1099852ac1126e8aebcdf61"
server_key_iv: "fcc9540e72c10ca9387847c192eedd30"
server_key_version: "encryptionkey@0"
I20260923 20:22:23.565896 22410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "32e2d393000c403880a2519dcc9ffdc0"
format_stamp: "Formatted at 2026-09-23 20:22:23 on dist-test-slave-wmd0"
server_key: "1a00372ab1099852ac1126e8aebcdf61"
server_key_iv: "fcc9540e72c10ca9387847c192eedd30"
server_key_version: "encryptionkey@0"
I20260923 20:22:23.570926 22410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:22:23.574334 22425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:23.575771 22410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:23.575966 22410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "32e2d393000c403880a2519dcc9ffdc0"
format_stamp: "Formatted at 2026-09-23 20:22:23 on dist-test-slave-wmd0"
server_key: "1a00372ab1099852ac1126e8aebcdf61"
server_key_iv: "fcc9540e72c10ca9387847c192eedd30"
server_key_version: "encryptionkey@0"
I20260923 20:22:23.576193 22410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:23.597813 22410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:23.602087 22410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:23.602385 22410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:23.624334 22410 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:46083
I20260923 20:22:23.624347 22486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:46083 every 8 connection(s)
I20260923 20:22:23.625712 22410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:23.629491 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:23.634289 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 22410
I20260923 20:22:23.634516 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:23.635208 14847 external_mini_cluster.cc:1428] Setting key 302a1d009b23b278863b0cc28496f54b
I20260923 20:22:23.640105 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: Bootstrap starting.
I20260923 20:22:23.643867 22487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:23.645162 22487 log.cc:824] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:23.648664 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: No bootstrap required, opened a new log
Sep 23 20:22:23 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:23.656608 22487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e2d393000c403880a2519dcc9ffdc0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46083 } }
I20260923 20:22:23.656994 22487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:23.657142 22487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e2d393000c403880a2519dcc9ffdc0, State: Initialized, Role: FOLLOWER
I20260923 20:22:23.657811 22487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e2d393000c403880a2519dcc9ffdc0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46083 } }
I20260923 20:22:23.658128 22487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:23.658284 22487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:23.658473 22487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:23.659308 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:23.637259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.14.127.254:46083)
0923 20:22:23.638110 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:23.638121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:23.638166 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:22:23.639799 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:23.639819 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:23.640212 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:23.640591 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:23.642494 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:23.643906 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:23.645705 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:23.646832 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:23.646972 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:23.651750 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:23.651759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:23.651774 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:23.651824 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:23.655082 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:23.655775 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:23.655782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:23.655787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:23.655965 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:23.656346 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:23.656352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:23.656356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:23.656873 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:23.657180 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:23.657670 (+ 490us) server_negotiation.cc:299] Negotiation successful
0923 20:22:23.658098 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":413,"threads_started":1}
I20260923 20:22:23.660880 22487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e2d393000c403880a2519dcc9ffdc0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46083 } }
I20260923 20:22:23.661375 22487 leader_election.cc:302] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32e2d393000c403880a2519dcc9ffdc0; no voters:
I20260923 20:22:23.662010 22487 leader_election.cc:288] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:23.662111 22492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:23.662889 22492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [term 1 LEADER]: Becoming Leader. State: Replica: 32e2d393000c403880a2519dcc9ffdc0, State: Running, Role: LEADER
I20260923 20:22:23.663535 22492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e2d393000c403880a2519dcc9ffdc0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46083 } }
I20260923 20:22:23.664318 22487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:23.666399 22494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32e2d393000c403880a2519dcc9ffdc0. Latest consensus state: current_term: 1 leader_uuid: "32e2d393000c403880a2519dcc9ffdc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e2d393000c403880a2519dcc9ffdc0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46083 } } }
I20260923 20:22:23.666826 22494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:23.666538 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32e2d393000c403880a2519dcc9ffdc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e2d393000c403880a2519dcc9ffdc0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46083 } } }
I20260923 20:22:23.666947 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:23.670428 22499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:23.675832 22499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:23.684892 22499 catalog_manager.cc:1347] Generated new cluster ID: 045d93fbc4014f02a4f3d1c087f0e46d
I20260923 20:22:23.685024 22499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:23.698766 22499 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:23.700136 22499 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:23.714111 22499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: Generated new TSK 0
I20260923 20:22:23.715094 22499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab.
Entry for principal oryx/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab.
I20260923 20:22:23.778815 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46083
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab
--principal=oryx/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:24.109014 22513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:24.109359 22513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:24.109512 22513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:24.116685 22513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:24.116861 22513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:24.117014 22513 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:22:24.117154 22513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:24.127151 22513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab
--principal=oryx/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:46083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:24.128553 22513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:24.129906 22513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:24.143127 22518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:24.143126 22519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:24.143250 22521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:24.143867 22513 server_base.cc:1034] running on GCE node
I20260923 20:22:24.144554 22513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:24.145486 22513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:24.146724 22513 hybrid_clock.cc:648] HybridClock initialized: now 1790194944146667 us; error 36 us; skew 500 ppm
I20260923 20:22:24.147053 22513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:24 dist-test-slave-wmd0 krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:24.156195 22513 init.cc:377] Logged in from keytab as oryx/127.14.127.193@KRBTEST.COM (short username oryx)
I20260923 20:22:24.157675 22513 webserver.cc:460] Webserver started at http://127.14.127.193:44081/ using document root <none> and password file <none>
I20260923 20:22:24.158404 22513 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:24.158538 22513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:24.158892 22513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:24.161854 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "a956441541e9437e8902b2b0239c7734"
format_stamp: "Formatted at 2026-09-23 20:22:24 on dist-test-slave-wmd0"
server_key: "e1360a35280a46458f224e28b04ec3a9"
server_key_iv: "faeda2a5d6ebbc72c81ead133bc3e31a"
server_key_version: "encryptionkey@0"
I20260923 20:22:24.162714 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "a956441541e9437e8902b2b0239c7734"
format_stamp: "Formatted at 2026-09-23 20:22:24 on dist-test-slave-wmd0"
server_key: "e1360a35280a46458f224e28b04ec3a9"
server_key_iv: "faeda2a5d6ebbc72c81ead133bc3e31a"
server_key_version: "encryptionkey@0"
I20260923 20:22:24.168184 22513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:22:24.172132 22528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:24.173511 22513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:24.173687 22513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "a956441541e9437e8902b2b0239c7734"
format_stamp: "Formatted at 2026-09-23 20:22:24 on dist-test-slave-wmd0"
server_key: "e1360a35280a46458f224e28b04ec3a9"
server_key_iv: "faeda2a5d6ebbc72c81ead133bc3e31a"
server_key_version: "encryptionkey@0"
I20260923 20:22:24.173924 22513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:24.190478 22513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:24.194933 22513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:24.195216 22513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:24.198007 22513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:24.198115 22513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:24.198248 22513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:24.198329 22513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:22:24 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:24.221402 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:24.198425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:53837 (local address 127.14.127.254:46083)
0923 20:22:24.198954 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:24.198963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:24.200111 (+ 1148us) server_negotiation.cc:407] Connection header received
0923 20:22:24.202161 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:24.202168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:24.202297 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:24.202438 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:24.204510 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.205570 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.208322 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.208947 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.209086 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:24.213524 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:24.213528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:24.213531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:24.213562 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:24.217341 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:24.218404 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:24.218408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:24.218410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:24.218521 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:24.219147 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:24.219152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:24.219155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:24.219593 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:24.219741 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:24.220545 (+ 804us) server_negotiation.cc:299] Negotiation successful
0923 20:22:24.220878 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":279,"threads_started":1}
I20260923 20:22:24.221890 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:24.199161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46083 (local address 127.14.127.193:53837)
0923 20:22:24.199954 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:24.199994 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:24.201810 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:24.202669 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:24.202686 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:24.203331 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:24.204319 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:24.204345 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.205762 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.205770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:24.208171 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:24.208183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.209190 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.209199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:24.209459 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:24.210131 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:24.210154 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:24.213292 (+ 3138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:24.217540 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:24.217555 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:24.217572 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:24.218238 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:24.218706 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:24.218713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:24.218718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:24.219006 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:24.219802 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:24.219807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:24.220125 (+ 318us) client_negotiation.cc:769] Sending connection context
0923 20:22:24.220352 (+ 227us) client_negotiation.cc:240] Negotiation successful
0923 20:22:24.220720 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":316,"threads_started":1}
I20260923 20:22:24.244921 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:24.228085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:47795 (local address 127.14.127.254:46083)
0923 20:22:24.228271 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:24.228279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:24.228912 (+ 633us) server_negotiation.cc:407] Connection header received
0923 20:22:24.229208 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:24.229215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:24.229321 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:24.229441 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:24.231209 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.232253 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.234870 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.235676 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.235828 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:24.238477 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:24.238482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:24.238485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:24.238513 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:24.241422 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:24.242415 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:24.242422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:24.242426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:24.242601 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:24.243131 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:24.243136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:24.243141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:24.243439 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:24.243557 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:24.244217 (+ 660us) server_negotiation.cc:299] Negotiation successful
0923 20:22:24.244496 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:22:24.245357 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:24.228151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46083 (local address 127.14.127.193:47795)
0923 20:22:24.228797 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:24.228827 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:24.229036 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:24.229637 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:24.229648 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:24.230410 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:24.231031 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:24.231048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.232410 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.232421 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:24.234698 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:24.234721 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.235914 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.235920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:24.236184 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:24.236965 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:24.236995 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:24.238289 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:24.241568 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:24.241577 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:24.241587 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:24.242271 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:24.242741 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:24.242749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:24.242755 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:24.242995 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:24.243576 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:24.243583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:24.243869 (+ 286us) client_negotiation.cc:769] Sending connection context
0923 20:22:24.244065 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 20:22:24.244552 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":342,"threads_started":1}
I20260923 20:22:24.248849 22513 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:42177
I20260923 20:22:24.248898 22648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:42177 every 8 connection(s)
I20260923 20:22:24.250283 22513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:24.252422 22649 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46083
I20260923 20:22:24.252650 22649 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:24.253293 22649 heartbeater.cc:511] Master 127.14.127.254:46083 requested a full tablet report, sending...
I20260923 20:22:24.253536 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 22513
I20260923 20:22:24.253731 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:24.254384 14847 external_mini_cluster.cc:1428] Setting key cb1c201f02206c6fa50864029a64e983
I20260923 20:22:24.255519 22442 ts_manager.cc:194] Registered new tserver with Master: a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177)
I20260923 20:22:24.257339 22442 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:53837
WARNING: no policy specified for oryx/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab.
Entry for principal oryx/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab.
I20260923 20:22:24.302966 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46083
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab
--principal=oryx/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:24.621114 22653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:24.621392 22653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:24.621622 22653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:24.629040 22653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:24.629184 22653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:24.629335 22653 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:22:24.629483 22653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:24.639767 22653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab
--principal=oryx/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:46083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:24.641012 22653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:24.642364 22653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:24.655514 22659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:24.655529 22661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:24.655443 22658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:24.656948 22653 server_base.cc:1034] running on GCE node
I20260923 20:22:24.657609 22653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:24.658715 22653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:24.660000 22653 hybrid_clock.cc:648] HybridClock initialized: now 1790194944659904 us; error 67 us; skew 500 ppm
I20260923 20:22:24.660326 22653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:24 dist-test-slave-wmd0 krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:24.663769 22653 init.cc:377] Logged in from keytab as oryx/127.14.127.194@KRBTEST.COM (short username oryx)
I20260923 20:22:24.665328 22653 webserver.cc:460] Webserver started at http://127.14.127.194:33015/ using document root <none> and password file <none>
I20260923 20:22:24.666115 22653 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:24.666247 22653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:24.666677 22653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:24.669991 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "12e8c3fd82504946a1dc1d006d73be17"
format_stamp: "Formatted at 2026-09-23 20:22:24 on dist-test-slave-wmd0"
server_key: "df7ab388cfb28cca3a3eebd3f66ca48f"
server_key_iv: "fed08da310ad1f03fb72860d1e6535c4"
server_key_version: "encryptionkey@0"
I20260923 20:22:24.671027 22653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "12e8c3fd82504946a1dc1d006d73be17"
format_stamp: "Formatted at 2026-09-23 20:22:24 on dist-test-slave-wmd0"
server_key: "df7ab388cfb28cca3a3eebd3f66ca48f"
server_key_iv: "fed08da310ad1f03fb72860d1e6535c4"
server_key_version: "encryptionkey@0"
I20260923 20:22:24.676666 22653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:22:24.680819 22668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:24.682255 22653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:22:24.682489 22653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "12e8c3fd82504946a1dc1d006d73be17"
format_stamp: "Formatted at 2026-09-23 20:22:24 on dist-test-slave-wmd0"
server_key: "df7ab388cfb28cca3a3eebd3f66ca48f"
server_key_iv: "fed08da310ad1f03fb72860d1e6535c4"
server_key_version: "encryptionkey@0"
I20260923 20:22:24.682792 22653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:24.699221 22653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:24.703559 22653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:24.703840 22653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:24.706664 22653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:24.706775 22653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 20:22:24.706965 22653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:24.707043 22653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:22:24 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:24.730441 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:24.707135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:60079 (local address 127.14.127.254:46083)
0923 20:22:24.707344 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:24.707350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:24.708770 (+ 1420us) server_negotiation.cc:407] Connection header received
0923 20:22:24.710875 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:24.710881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:24.711009 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:24.711126 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:24.713434 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.714544 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.717294 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.717890 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.718046 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:24.722989 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:24.722997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:24.723006 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:24.723040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:24.726194 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:24.727070 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:24.727076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:24.727079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:24.727206 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:24.727856 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:24.727860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:24.727862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:24.728252 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:24.728422 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:24.729573 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0923 20:22:24.729920 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:22:24.731153 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:24.707827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46083 (local address 127.14.127.194:60079)
0923 20:22:24.708602 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:24.708645 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:24.710535 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:24.711374 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:24.711385 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:24.712210 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:24.713190 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:24.713218 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.714759 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.714764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:24.717137 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:24.717153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.718175 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.718182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:24.718407 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:24.719279 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:24.719329 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:24.722691 (+ 3362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:24.726344 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:24.726354 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:24.726378 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:24.726907 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:24.727378 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:24.727383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:24.727387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:24.727716 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:24.728490 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:24.728496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:24.729062 (+ 566us) client_negotiation.cc:769] Sending connection context
0923 20:22:24.729417 (+ 355us) client_negotiation.cc:240] Negotiation successful
0923 20:22:24.729887 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":343,"threads_started":1}
I20260923 20:22:24.753158 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:24.738385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:55293 (local address 127.14.127.254:46083)
0923 20:22:24.738634 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:24.738642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:24.739358 (+ 716us) server_negotiation.cc:407] Connection header received
0923 20:22:24.739540 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:24.739543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:24.739603 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:24.739683 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:24.741147 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.741975 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.744229 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.744817 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.744967 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:24.746876 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:24.746913 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:24.746917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:24.746956 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:24.749964 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:24.750644 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:24.750649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:24.750652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:24.750776 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:24.751279 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:24.751287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:24.751293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:24.751816 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:24.752282 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:24.752575 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 20:22:24.752796 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:22:24.753257 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:24.738340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46083 (local address 127.14.127.194:55293)
0923 20:22:24.739212 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:24.739237 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:24.739410 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:24.739752 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:24.739759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:24.740520 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:24.740992 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:24.741008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.742159 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.742166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:24.744005 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:24.744025 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:24.745024 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:24.745031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:24.745216 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:24.745900 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:24.745917 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:24.746674 (+ 757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:24.750085 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:24.750092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:24.750096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:24.750489 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:24.750899 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:24.750906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:24.750910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:24.751125 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:24.752012 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:24.752021 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:24.752296 (+ 275us) client_negotiation.cc:769] Sending connection context
0923 20:22:24.752472 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 20:22:24.752797 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":473,"threads_started":1}
I20260923 20:22:24.766579 22653 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:37311
I20260923 20:22:24.766652 22787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:37311 every 8 connection(s)
I20260923 20:22:24.767839 22653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:24.769999 22788 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46083
I20260923 20:22:24.770233 22788 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:24.770936 22788 heartbeater.cc:511] Master 127.14.127.254:46083 requested a full tablet report, sending...
I20260923 20:22:24.772205 22442 ts_manager.cc:194] Registered new tserver with Master: 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311)
I20260923 20:22:24.773044 22442 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:60079
I20260923 20:22:24.773380 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 22653
I20260923 20:22:24.773555 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:24.774121 14847 external_mini_cluster.cc:1428] Setting key f55099a2e598a6e01014c1f9dc468ea5
WARNING: no policy specified for oryx/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab.
Entry for principal oryx/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab.
I20260923 20:22:24.822525 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46083
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab
--principal=oryx/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:25.137964 22792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:25.138291 22792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:25.138471 22792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:25.145606 22792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:25.145761 22792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:25.145928 22792 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:22:25.146073 22792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:25.155616 22792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab
--principal=oryx/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.127.254:46083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:25.156877 22792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:25.158167 22792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:25.171257 22800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:25.171373 22797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:25.171442 22798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:25.172009 22792 server_base.cc:1034] running on GCE node
I20260923 20:22:25.172628 22792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:25.173491 22792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:25.174727 22792 hybrid_clock.cc:648] HybridClock initialized: now 1790194945174663 us; error 40 us; skew 500 ppm
I20260923 20:22:25.175024 22792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:25 dist-test-slave-wmd0 krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194945, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:25.178090 22792 init.cc:377] Logged in from keytab as oryx/127.14.127.195@KRBTEST.COM (short username oryx)
I20260923 20:22:25.179555 22792 webserver.cc:460] Webserver started at http://127.14.127.195:38907/ using document root <none> and password file <none>
I20260923 20:22:25.180302 22792 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:25.180429 22792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:25.180816 22792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:25.183712 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "171094e880fc4392b44cb74916ddded2"
format_stamp: "Formatted at 2026-09-23 20:22:25 on dist-test-slave-wmd0"
server_key: "ba0de0fe12ecb74834888493f31b5c3a"
server_key_iv: "b591295d17004d1adeb131961d1d427d"
server_key_version: "encryptionkey@0"
I20260923 20:22:25.184475 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "171094e880fc4392b44cb74916ddded2"
format_stamp: "Formatted at 2026-09-23 20:22:25 on dist-test-slave-wmd0"
server_key: "ba0de0fe12ecb74834888493f31b5c3a"
server_key_iv: "b591295d17004d1adeb131961d1d427d"
server_key_version: "encryptionkey@0"
I20260923 20:22:25.189572 22792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260923 20:22:25.192889 22807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:25.194137 22792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:22:25.194293 22792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "171094e880fc4392b44cb74916ddded2"
format_stamp: "Formatted at 2026-09-23 20:22:25 on dist-test-slave-wmd0"
server_key: "ba0de0fe12ecb74834888493f31b5c3a"
server_key_iv: "b591295d17004d1adeb131961d1d427d"
server_key_version: "encryptionkey@0"
I20260923 20:22:25.194489 22792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:25.224651 22792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:25.229043 22792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:25.229321 22792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:25.232101 22792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:25.232195 22792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:25.232348 22792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:25.232432 22792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:22:25 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194945, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:25.255798 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.232656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:43773 (local address 127.14.127.254:46083)
0923 20:22:25.232884 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.232890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.234339 (+ 1449us) server_negotiation.cc:407] Connection header received
0923 20:22:25.236582 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.236590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.236686 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.236797 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:25.238200 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.239314 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.240895 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.241635 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.241762 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.246927 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:25.246935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:25.246943 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:25.246993 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:25.251613 (+ 4620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:25.252852 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:25.252861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:25.252866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:25.253038 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:25.253702 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:25.253710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:25.253716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:25.254130 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:25.254322 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.255202 (+ 880us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.255443 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:22:25.257200 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.233328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46083 (local address 127.14.127.195:43773)
0923 20:22:25.234172 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.234213 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.236314 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.237029 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.237037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.237490 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:25.238022 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.238038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.239481 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.239496 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.240721 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.240732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.241890 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.241898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.242272 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.243067 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:25.243102 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:25.246663 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:25.251855 (+ 5192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:25.251864 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:25.251882 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:25.252655 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:25.253195 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:25.253200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:25.253208 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:25.253505 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:25.254295 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:25.254303 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:25.254717 (+ 414us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.255066 (+ 349us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.255594 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":333,"threads_started":1}
I20260923 20:22:25.260107 22649 heartbeater.cc:503] Master 127.14.127.254:46083 was elected leader, sending a full tablet report...
I20260923 20:22:25.281028 22792 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:46703
I20260923 20:22:25.281042 22926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:46703 every 8 connection(s)
I20260923 20:22:25.281438 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.265471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:55119 (local address 127.14.127.254:46083)
0923 20:22:25.265702 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.265711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.266267 (+ 556us) server_negotiation.cc:407] Connection header received
0923 20:22:25.266497 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.266506 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.266589 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.266737 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:25.267651 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.268942 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.270296 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.271102 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.271315 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.273914 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:25.273923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:25.273930 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:25.273974 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:25.278034 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:25.278997 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:25.279002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:25.279005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:25.279176 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:25.279596 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:25.279604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:25.279610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:25.280032 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:25.280284 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.280848 (+ 564us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.281083 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:22:25.281565 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.265506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46083 (local address 127.14.127.195:55119)
0923 20:22:25.266146 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.266170 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.266338 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.266756 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.266761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.267180 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:25.267502 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.267512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.269104 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.269109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.270155 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.270166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.271395 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.271403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.271657 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.272389 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:25.272406 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:25.273671 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:25.278278 (+ 4607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:25.278288 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:25.278293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:25.278868 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:25.279357 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:25.279360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:25.279363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:25.279479 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:25.280230 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:25.280235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:25.280497 (+ 262us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.280694 (+ 197us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.281133 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":344,"threads_started":1}
I20260923 20:22:25.282718 22792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:25.285192 22927 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46083
I20260923 20:22:25.285897 22927 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:25.286588 22927 heartbeater.cc:511] Master 127.14.127.254:46083 requested a full tablet report, sending...
I20260923 20:22:25.288234 22442 ts_manager.cc:194] Registered new tserver with Master: 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703)
I20260923 20:22:25.288998 22442 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:43773
I20260923 20:22:25.292393 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 22792
I20260923 20:22:25.292568 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:25.293016 14847 external_mini_cluster.cc:1428] Setting key 9027cad438c69d621ea2aeb9d9317610
I20260923 20:22:25.296039 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:22:25 dist-test-slave-wmd0 krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:22:25 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:25 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:22:25.343868 22536 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:25.333573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.14.127.254:46083)
0923 20:22:25.333749 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.333753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.334175 (+ 422us) server_negotiation.cc:407] Connection header received
0923 20:22:25.334386 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.334390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.334480 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.334569 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:25.335885 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.336617 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.337790 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.338197 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.338296 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.343527 (+ 5231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58180: BlockingRecv error: recv got EOF from 127.0.0.1:58180 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 20:22:25 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:25.359465 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.347060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.14.127.254:46083)
0923 20:22:25.347217 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.347221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.347585 (+ 364us) server_negotiation.cc:407] Connection header received
0923 20:22:25.347706 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.347709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.347773 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.347854 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:25.348882 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.349715 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.350923 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.351327 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.351424 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.354784 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:25.354788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:25.354791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:25.354821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:25.357127 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:25.357819 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:25.357823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:25.357825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:25.357939 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:25.358239 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:25.358242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:25.358244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:25.358501 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:25.358773 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.358989 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.359139 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:22:25.363168 22442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:25.370297 22442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:25.394546 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.382498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:37311 (local address 127.0.0.1:57734)
0923 20:22:25.383265 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.383348 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.383521 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.384798 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.384808 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.384856 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:25.385370 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.385386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.387961 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.387970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.390934 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.390954 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.393255 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.393265 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.393575 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.393662 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.393785 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.394032 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":364,"threads_started":1}
I20260923 20:22:25.397174 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.382671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.14.127.194:37311)
0923 20:22:25.384080 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.384090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.384120 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:22:25.384218 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.384227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.384512 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.385614 (+ 1102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:25.385834 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.387785 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.391184 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.392993 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.396171 (+ 3178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.396450 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.396538 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.396769 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1236,"thread_start_us":1120,"threads_started":1}
I20260923 20:22:25.403036 22718 tablet_service.cc:1467] Processing CreateTablet for tablet c3d142e37da94b088799b438c6219208 (DEFAULT_TABLE table=test-table [id=43e14be93fc6483184a1f33fface46a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:25.404611 22718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d142e37da94b088799b438c6219208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.405210 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.383170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:42177 (local address 127.0.0.1:48304)
0923 20:22:25.395162 (+ 11992us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.395186 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.395322 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.399245 (+ 3923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.399252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.399278 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:25.399475 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.399485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.401318 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.401324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.403087 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.403098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.404504 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.404509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.404722 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.404752 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.404815 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.404921 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11825,"thread_start_us":13074,"threads_started":1}
I20260923 20:22:25.405269 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.383645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46703 (local address 127.0.0.1:43086)
0923 20:22:25.396052 (+ 12407us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.396076 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.396589 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.397911 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.397919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.397955 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:25.398418 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.398429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.400337 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.400343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.402532 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.402552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.404370 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.404379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.404554 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.404599 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.404704 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.404844 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12231,"mutex_wait_us":4,"thread_start_us":14316,"threads_started":1}
I20260923 20:22:25.406302 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.383379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.14.127.193:42177)
0923 20:22:25.398583 (+ 15204us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.398590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.398646 (+ 56us) server_negotiation.cc:407] Connection header received
0923 20:22:25.398769 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.398779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.399034 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.399240 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:25.399640 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.401117 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.403243 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.404330 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.405580 (+ 1250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.405769 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.405842 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.406004 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15031,"thread_start_us":14888,"threads_started":1}
I20260923 20:22:25.406503 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.384248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.14.127.195:46703)
0923 20:22:25.395979 (+ 11731us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.395991 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.396208 (+ 217us) server_negotiation.cc:407] Connection header received
0923 20:22:25.396841 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.396853 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.397203 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.397410 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:25.398645 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.400142 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.402727 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.404142 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.404353 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.404514 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.404765 (+ 251us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.406179 (+ 1414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11533,"thread_start_us":11346,"threads_started":1}
I20260923 20:22:25.409885 22857 tablet_service.cc:1467] Processing CreateTablet for tablet c3d142e37da94b088799b438c6219208 (DEFAULT_TABLE table=test-table [id=43e14be93fc6483184a1f33fface46a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:25.409885 22579 tablet_service.cc:1467] Processing CreateTablet for tablet c3d142e37da94b088799b438c6219208 (DEFAULT_TABLE table=test-table [id=43e14be93fc6483184a1f33fface46a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:25.411319 22579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d142e37da94b088799b438c6219208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.411399 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d142e37da94b088799b438c6219208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.417771 22951 tablet_bootstrap.cc:492] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Bootstrap starting.
I20260923 20:22:25.421685 22951 tablet_bootstrap.cc:654] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.422802 22951 log.cc:824] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:25.427232 22951 tablet_bootstrap.cc:492] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: No bootstrap required, opened a new log
I20260923 20:22:25.427211 22954 tablet_bootstrap.cc:492] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Bootstrap starting.
I20260923 20:22:25.427536 22951 ts_tablet_manager.cc:1397] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260923 20:22:25.429970 22954 tablet_bootstrap.cc:654] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.430994 22954 log.cc:824] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:25.431157 22953 tablet_bootstrap.cc:492] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Bootstrap starting.
I20260923 20:22:25.434159 22954 tablet_bootstrap.cc:492] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: No bootstrap required, opened a new log
I20260923 20:22:25.434429 22954 ts_tablet_manager.cc:1397] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260923 20:22:25.434506 22953 tablet_bootstrap.cc:654] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.435456 22951 raft_consensus.cc:348] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.435853 22951 raft_consensus.cc:374] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.435853 22953 log.cc:824] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:25.435978 22951 raft_consensus.cc:729] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c3fd82504946a1dc1d006d73be17, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.436575 22951 consensus_queue.cc:260] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.437593 22788 heartbeater.cc:503] Master 127.14.127.254:46083 was elected leader, sending a full tablet report...
I20260923 20:22:25.438316 22951 ts_tablet_manager.cc:1428] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260923 20:22:25.441295 22953 tablet_bootstrap.cc:492] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: No bootstrap required, opened a new log
I20260923 20:22:25.441619 22953 ts_tablet_manager.cc:1397] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260923 20:22:25.445080 22954 raft_consensus.cc:348] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.445454 22954 raft_consensus.cc:374] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.445566 22954 raft_consensus.cc:729] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171094e880fc4392b44cb74916ddded2, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.446166 22954 consensus_queue.cc:260] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.447317 22927 heartbeater.cc:503] Master 127.14.127.254:46083 was elected leader, sending a full tablet report...
I20260923 20:22:25.447934 22954 ts_tablet_manager.cc:1428] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260923 20:22:25.449993 22953 raft_consensus.cc:348] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.450685 22953 raft_consensus.cc:374] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.450796 22953 raft_consensus.cc:729] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a956441541e9437e8902b2b0239c7734, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.451380 22953 consensus_queue.cc:260] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.453330 22953 ts_tablet_manager.cc:1428] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
W20260923 20:22:25.502854 22650 tablet.cc:2369] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:22:25.519994 22789 tablet.cc:2369] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:22:25.536273 22928 tablet.cc:2369] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:22:25.649432 22957 raft_consensus.cc:482] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:25.649775 22957 raft_consensus.cc:504] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.651638 22957 leader_election.cc:288] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703), a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177)
I20260923 20:22:25.658500 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.652294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:41669 (local address 127.14.127.195:46703)
0923 20:22:25.652536 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.652544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.653141 (+ 597us) server_negotiation.cc:407] Connection header received
0923 20:22:25.653387 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.653392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.653452 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.653550 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:25.654116 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.654924 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.656773 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.657521 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.657714 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.657791 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.658102 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.658237 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:22:25.658653 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.652128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:48459 (local address 127.14.127.193:42177)
0923 20:22:25.652369 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.652378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.653386 (+ 1008us) server_negotiation.cc:407] Connection header received
0923 20:22:25.653587 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.653591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.653652 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.653748 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:25.654360 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.655109 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.656936 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.657713 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.657904 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.657978 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.658239 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.658377 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:22:25.658715 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.651944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46703 (local address 127.14.127.194:41669)
0923 20:22:25.653017 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.653034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.653222 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.653576 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.653580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.653605 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:25.653960 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.653972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.655056 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.655064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.656607 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.656620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.657735 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.657741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.657934 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.657960 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.658073 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.658204 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":2,"thread_start_us":989,"threads_started":1}
I20260923 20:22:25.659124 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.652157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:42177 (local address 127.14.127.194:48459)
0923 20:22:25.653258 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.653274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.653430 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.653796 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.653801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.653826 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:25.654181 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.654190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.655254 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.655260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.656786 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.656798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.657934 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.657940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.658088 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.658110 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.658215 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.658342 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":42,"thread_start_us":403,"threads_started":1}
I20260923 20:22:25.659873 22880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d142e37da94b088799b438c6219208" candidate_uuid: "12e8c3fd82504946a1dc1d006d73be17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171094e880fc4392b44cb74916ddded2" is_pre_election: true
I20260923 20:22:25.660127 22604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d142e37da94b088799b438c6219208" candidate_uuid: "12e8c3fd82504946a1dc1d006d73be17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956441541e9437e8902b2b0239c7734" is_pre_election: true
I20260923 20:22:25.660521 22880 raft_consensus.cc:2454] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c3fd82504946a1dc1d006d73be17 in term 0.
I20260923 20:22:25.660652 22604 raft_consensus.cc:2454] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c3fd82504946a1dc1d006d73be17 in term 0.
I20260923 20:22:25.661321 22670 leader_election.cc:302] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12e8c3fd82504946a1dc1d006d73be17, 171094e880fc4392b44cb74916ddded2, a956441541e9437e8902b2b0239c7734; no voters:
I20260923 20:22:25.661788 22957 raft_consensus.cc:2781] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:25.661945 22957 raft_consensus.cc:482] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:25.662071 22957 raft_consensus.cc:3037] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:25.664441 22957 raft_consensus.cc:504] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.665125 22957 leader_election.cc:288] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [CANDIDATE]: Term 1 election: Requested vote from peers 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703), a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177)
I20260923 20:22:25.665504 22880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d142e37da94b088799b438c6219208" candidate_uuid: "12e8c3fd82504946a1dc1d006d73be17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171094e880fc4392b44cb74916ddded2"
I20260923 20:22:25.665669 22604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d142e37da94b088799b438c6219208" candidate_uuid: "12e8c3fd82504946a1dc1d006d73be17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956441541e9437e8902b2b0239c7734"
I20260923 20:22:25.665728 22880 raft_consensus.cc:3037] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:25.665895 22604 raft_consensus.cc:3037] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:25.668041 22880 raft_consensus.cc:2454] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c3fd82504946a1dc1d006d73be17 in term 1.
I20260923 20:22:25.668221 22604 raft_consensus.cc:2454] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c3fd82504946a1dc1d006d73be17 in term 1.
I20260923 20:22:25.668462 22671 leader_election.cc:302] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c3fd82504946a1dc1d006d73be17, 171094e880fc4392b44cb74916ddded2; no voters:
I20260923 20:22:25.668756 22957 raft_consensus.cc:2781] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:25.669209 22957 raft_consensus.cc:686] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c3fd82504946a1dc1d006d73be17, State: Running, Role: LEADER
I20260923 20:22:25.669685 22957 consensus_queue.cc:237] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } }
I20260923 20:22:25.674129 22441 catalog_manager.cc:5495] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194). New cstate: current_term: 1 leader_uuid: "12e8c3fd82504946a1dc1d006d73be17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } health_report { overall_health: UNKNOWN } } }
I20260923 20:22:25.706346 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.701284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.14.127.194:37311)
0923 20:22:25.701502 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.701507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.701526 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:22:25.701635 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.701639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.701712 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.701870 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:25.702221 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.703004 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.704302 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.704688 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.704837 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.705255 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:25.705768 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:25.705933 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.705995 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.706073 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:22:25.714301 22880 raft_consensus.cc:1261] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c3fd82504946a1dc1d006d73be17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:25.714301 22604 raft_consensus.cc:1261] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c3fd82504946a1dc1d006d73be17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:25.715410 22962 consensus_queue.cc:1035] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:25.715780 22957 consensus_queue.cc:1035] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:25.726785 22965 mvcc.cc:205] Tried to move back new op lower bound from 7332638497636089856 to 7332638497466740736. Current Snapshot: MvccSnapshot[applied={T|T < 7332638497636089856}]
I20260923 20:22:25.739522 22442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:25.739776 22442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:25.743304 22442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: Sending DeleteTablet for 3 replicas of tablet c3d142e37da94b088799b438c6219208
I20260923 20:22:25.744369 22718 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d142e37da94b088799b438c6219208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:25 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57734
I20260923 20:22:25.744603 22857 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d142e37da94b088799b438c6219208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:25 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43086
I20260923 20:22:25.744997 22579 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d142e37da94b088799b438c6219208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:25 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:48304
I20260923 20:22:25.745581 22975 tablet_replica.cc:330] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: stopping tablet replica
I20260923 20:22:25.746052 22975 raft_consensus.cc:2229] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:25.746006 22442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:25.746690 22442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:25.746884 22975 raft_consensus.cc:2258] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:25.748916 22976 tablet_replica.cc:330] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: stopping tablet replica
I20260923 20:22:25.749274 22975 ts_tablet_manager.cc:1905] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:25.749310 22976 raft_consensus.cc:2229] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:25.749930 22976 raft_consensus.cc:2258] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:25.750662 22977 tablet_replica.cc:330] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: stopping tablet replica
I20260923 20:22:25.753173 22976 ts_tablet_manager.cc:1905] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:25.754345 22579 tablet_service.cc:1467] Processing CreateTablet for tablet a05e2ba2d0fd495f806919083f23d1f2 (DEFAULT_TABLE table=test-table [id=731cb18d09224f57afcc8bff16ca7a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:25.754806 22579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05e2ba2d0fd495f806919083f23d1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.754989 22857 tablet_service.cc:1467] Processing CreateTablet for tablet a05e2ba2d0fd495f806919083f23d1f2 (DEFAULT_TABLE table=test-table [id=731cb18d09224f57afcc8bff16ca7a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:25.755820 22718 tablet_service.cc:1467] Processing CreateTablet for tablet a05e2ba2d0fd495f806919083f23d1f2 (DEFAULT_TABLE table=test-table [id=731cb18d09224f57afcc8bff16ca7a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:25.756242 22718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05e2ba2d0fd495f806919083f23d1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.760602 22976 ts_tablet_manager.cc:1918] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:25.760772 22976 log.cc:1196] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/c3d142e37da94b088799b438c6219208
I20260923 20:22:25.761389 22976 ts_tablet_manager.cc:1939] T c3d142e37da94b088799b438c6219208 P 12e8c3fd82504946a1dc1d006d73be17: Deleting consensus metadata
I20260923 20:22:25.761819 22953 tablet_bootstrap.cc:492] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: Bootstrap starting.
I20260923 20:22:25.762373 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05e2ba2d0fd495f806919083f23d1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.763417 22953 tablet_bootstrap.cc:654] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.764400 22951 tablet_bootstrap.cc:492] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: Bootstrap starting.
I20260923 20:22:25.764957 22977 raft_consensus.cc:2229] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:25.765997 22951 tablet_bootstrap.cc:654] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.766582 22977 raft_consensus.cc:2258] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:25.769800 22953 tablet_bootstrap.cc:492] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: No bootstrap required, opened a new log
I20260923 20:22:25.769966 22953 ts_tablet_manager.cc:1397] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260923 20:22:25.770020 22954 tablet_bootstrap.cc:492] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: Bootstrap starting.
I20260923 20:22:25.770733 22953 raft_consensus.cc:348] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.770972 22953 raft_consensus.cc:374] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.770965 22428 catalog_manager.cc:4841] TS 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311): tablet c3d142e37da94b088799b438c6219208 (table test-table [id=43e14be93fc6483184a1f33fface46a6]) successfully deleted
I20260923 20:22:25.771070 22953 raft_consensus.cc:729] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a956441541e9437e8902b2b0239c7734, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.771694 22954 tablet_bootstrap.cc:654] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.773844 22975 ts_tablet_manager.cc:1918] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:25.774056 22975 log.cc:1196] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/c3d142e37da94b088799b438c6219208
I20260923 20:22:25.775404 22977 ts_tablet_manager.cc:1905] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:25.777143 22953 consensus_queue.cc:260] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.777640 22975 ts_tablet_manager.cc:1939] T c3d142e37da94b088799b438c6219208 P 171094e880fc4392b44cb74916ddded2: Deleting consensus metadata
I20260923 20:22:25.779755 22428 catalog_manager.cc:4841] TS 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703): tablet c3d142e37da94b088799b438c6219208 (table test-table [id=43e14be93fc6483184a1f33fface46a6]) successfully deleted
I20260923 20:22:25.780335 22953 ts_tablet_manager.cc:1428] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260923 20:22:25.781769 22954 tablet_bootstrap.cc:492] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: No bootstrap required, opened a new log
I20260923 20:22:25.781935 22954 ts_tablet_manager.cc:1397] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20260923 20:22:25.782670 22954 raft_consensus.cc:348] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.782917 22954 raft_consensus.cc:374] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.783007 22954 raft_consensus.cc:729] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171094e880fc4392b44cb74916ddded2, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.783248 22954 consensus_queue.cc:260] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.783768 22954 ts_tablet_manager.cc:1428] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:22:25.787226 22977 ts_tablet_manager.cc:1918] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:25.787400 22977 log.cc:1196] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/c3d142e37da94b088799b438c6219208
I20260923 20:22:25.787988 22977 ts_tablet_manager.cc:1939] T c3d142e37da94b088799b438c6219208 P a956441541e9437e8902b2b0239c7734: Deleting consensus metadata
I20260923 20:22:25.789147 22951 tablet_bootstrap.cc:492] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: No bootstrap required, opened a new log
I20260923 20:22:25.789314 22951 ts_tablet_manager.cc:1397] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20260923 20:22:25.789903 22427 catalog_manager.cc:4841] TS a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177): tablet c3d142e37da94b088799b438c6219208 (table test-table [id=43e14be93fc6483184a1f33fface46a6]) successfully deleted
I20260923 20:22:25.790100 22951 raft_consensus.cc:348] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.790336 22951 raft_consensus.cc:374] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.790452 22951 raft_consensus.cc:729] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c3fd82504946a1dc1d006d73be17, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.790736 22951 consensus_queue.cc:260] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.791303 22951 ts_tablet_manager.cc:1428] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:22:25.854053 22958 raft_consensus.cc:482] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:25.854262 22958 raft_consensus.cc:504] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.855834 22958 leader_election.cc:288] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177), 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311)
I20260923 20:22:25.863358 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.855987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:42177 (local address 127.14.127.195:58131)
0923 20:22:25.856931 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.856956 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.857172 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.857575 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.857580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.857604 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:25.857844 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.857862 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.859382 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.859387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.861136 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.861148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.862484 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.862490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.862698 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.862736 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.862833 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.862982 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":46,"thread_start_us":682,"threads_started":1}
I20260923 20:22:25.864137 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.856333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:58131 (local address 127.14.127.193:42177)
0923 20:22:25.856497 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.856503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.857059 (+ 556us) server_negotiation.cc:407] Connection header received
0923 20:22:25.857375 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.857383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.857458 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.857557 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:25.858017 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.859191 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.861281 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.862310 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.862471 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.862542 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.862888 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.863857 (+ 969us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:22:25.864524 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.856310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:58625 (local address 127.14.127.194:37311)
0923 20:22:25.856497 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.856504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.857274 (+ 770us) server_negotiation.cc:407] Connection header received
0923 20:22:25.857419 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.857424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.857503 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.857600 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:25.858318 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.859217 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.861865 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.863158 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.863307 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.863397 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.864076 (+ 679us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.864236 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:22:25.864912 22604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e2ba2d0fd495f806919083f23d1f2" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956441541e9437e8902b2b0239c7734" is_pre_election: true
I20260923 20:22:25.865185 22604 raft_consensus.cc:2454] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 0.
I20260923 20:22:25.865309 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.856081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:37311 (local address 127.14.127.195:58625)
0923 20:22:25.857170 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.857185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.857296 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.857726 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.857737 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.857761 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:25.858101 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.858110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.859405 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.859411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.861691 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.861707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.863675 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.863684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.863847 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.863877 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.864760 (+ 883us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.864880 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":3,"thread_start_us":853,"threads_started":1}
I20260923 20:22:25.865646 22809 leader_election.cc:302] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171094e880fc4392b44cb74916ddded2, a956441541e9437e8902b2b0239c7734; no voters:
I20260923 20:22:25.866089 22958 raft_consensus.cc:2781] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:25.866211 22958 raft_consensus.cc:482] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:25.866297 22958 raft_consensus.cc:3037] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:25.866233 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e2ba2d0fd495f806919083f23d1f2" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c3fd82504946a1dc1d006d73be17" is_pre_election: true
I20260923 20:22:25.866729 22743 raft_consensus.cc:2454] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 0.
I20260923 20:22:25.869498 22958 raft_consensus.cc:504] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.870198 22958 leader_election.cc:288] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 election: Requested vote from peers a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177), 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311)
I20260923 20:22:25.870666 22604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e2ba2d0fd495f806919083f23d1f2" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956441541e9437e8902b2b0239c7734"
I20260923 20:22:25.870744 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e2ba2d0fd495f806919083f23d1f2" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c3fd82504946a1dc1d006d73be17"
I20260923 20:22:25.870870 22604 raft_consensus.cc:3037] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:25.871001 22743 raft_consensus.cc:3037] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:25.873211 22743 raft_consensus.cc:2454] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 1.
I20260923 20:22:25.873617 22810 leader_election.cc:302] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c3fd82504946a1dc1d006d73be17, 171094e880fc4392b44cb74916ddded2; no voters:
I20260923 20:22:25.873720 22604 raft_consensus.cc:2454] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 1.
I20260923 20:22:25.873934 22958 raft_consensus.cc:2781] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:25.874449 22958 raft_consensus.cc:686] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 1 LEADER]: Becoming Leader. State: Replica: 171094e880fc4392b44cb74916ddded2, State: Running, Role: LEADER
I20260923 20:22:25.874863 22958 consensus_queue.cc:237] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.878108 22442 catalog_manager.cc:5495] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 171094e880fc4392b44cb74916ddded2 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } health_report { overall_health: HEALTHY } } }
Sep 23 20:22:25 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194943, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.254@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:25.930367 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.916969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.14.127.254:46083)
0923 20:22:25.917687 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:25.917691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:25.917717 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:22:25.917757 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:25.917760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:25.917819 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:25.917965 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:25.919124 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.919874 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.921050 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.921460 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.921600 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.925038 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:25.925043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:25.925046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:25.925072 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:25.927695 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:25.928369 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:25.928377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:25.928381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:25.928504 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:25.928872 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:25.928877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:25.928880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:25.929128 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:25.929278 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:25.929669 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 20:22:25.930010 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":583,"thread_start_us":477,"threads_started":1}
I20260923 20:22:25.930388 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:25.916854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46083 (local address 127.0.0.1:58202)
0923 20:22:25.917374 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:25.917401 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:25.917559 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:25.917988 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:25.917992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:25.918748 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:25.918967 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.918976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.920006 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.920012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.920912 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:25.920922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:25.921621 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:25.921627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:25.921816 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:25.922513 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:25.922531 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:25.924840 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:25.927835 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:25.927848 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:25.927856 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:25.928247 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:25.928612 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:25.928619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:25.928622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:25.928757 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:25.929254 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:25.929262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:25.929432 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 20:22:25.929595 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 20:22:25.929901 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":3,"thread_start_us":265,"threads_started":1}
I20260923 20:22:25.936017 22442 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:58202:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:22:25.937008 22442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:25.943732 22579 tablet_service.cc:1467] Processing CreateTablet for tablet 35d9bccfd0574581b2bc49e0e7850514 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5578094c8b044de87536a2a8a38690a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:22:25.944159 22718 tablet_service.cc:1467] Processing CreateTablet for tablet 35d9bccfd0574581b2bc49e0e7850514 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5578094c8b044de87536a2a8a38690a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:22:25.944348 22579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d9bccfd0574581b2bc49e0e7850514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.944489 22857 tablet_service.cc:1467] Processing CreateTablet for tablet 35d9bccfd0574581b2bc49e0e7850514 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5578094c8b044de87536a2a8a38690a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:22:25.944725 22718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d9bccfd0574581b2bc49e0e7850514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.945014 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d9bccfd0574581b2bc49e0e7850514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:25.951742 22953 tablet_bootstrap.cc:492] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: Bootstrap starting.
I20260923 20:22:25.951750 22951 tablet_bootstrap.cc:492] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: Bootstrap starting.
I20260923 20:22:25.953076 22951 tablet_bootstrap.cc:654] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.953387 22953 tablet_bootstrap.cc:654] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.955317 22951 tablet_bootstrap.cc:492] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: No bootstrap required, opened a new log
I20260923 20:22:25.955490 22951 ts_tablet_manager.cc:1397] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 20:22:25.956103 22954 tablet_bootstrap.cc:492] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: Bootstrap starting.
I20260923 20:22:25.956435 22951 raft_consensus.cc:348] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.956681 22951 raft_consensus.cc:374] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.956763 22951 raft_consensus.cc:729] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c3fd82504946a1dc1d006d73be17, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.957059 22951 consensus_queue.cc:260] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.957398 22953 tablet_bootstrap.cc:492] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: No bootstrap required, opened a new log
I20260923 20:22:25.957549 22953 ts_tablet_manager.cc:1397] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260923 20:22:25.957816 22954 tablet_bootstrap.cc:654] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:25.957523 22970 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:25.957935 22970 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:22:25.958235 22953 raft_consensus.cc:348] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.958460 22953 raft_consensus.cc:374] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.958529 22953 raft_consensus.cc:729] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a956441541e9437e8902b2b0239c7734, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.960038 22953 consensus_queue.cc:260] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.960551 22959 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:25.960876 22959 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:22:25.961001 22951 ts_tablet_manager.cc:1428] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260923 20:22:25.964609 22954 tablet_bootstrap.cc:492] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: No bootstrap required, opened a new log
I20260923 20:22:25.964783 22954 ts_tablet_manager.cc:1397] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260923 20:22:25.964818 22953 ts_tablet_manager.cc:1428] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260923 20:22:25.965514 22954 raft_consensus.cc:348] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.965772 22954 raft_consensus.cc:374] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:25.965875 22954 raft_consensus.cc:729] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171094e880fc4392b44cb74916ddded2, State: Initialized, Role: FOLLOWER
I20260923 20:22:25.966104 22954 consensus_queue.cc:260] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:25.966542 22958 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:25.966868 22958 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:22:25.967278 22954 ts_tablet_manager.cc:1428] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 20:22:26.145174 22958 raft_consensus.cc:482] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:26.145452 22958 raft_consensus.cc:504] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.146155 22958 leader_election.cc:288] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177), 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311)
I20260923 20:22:26.146720 22604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d9bccfd0574581b2bc49e0e7850514" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956441541e9437e8902b2b0239c7734" is_pre_election: true
I20260923 20:22:26.146847 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d9bccfd0574581b2bc49e0e7850514" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c3fd82504946a1dc1d006d73be17" is_pre_election: true
I20260923 20:22:26.147029 22604 raft_consensus.cc:2454] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 0.
I20260923 20:22:26.147126 22743 raft_consensus.cc:2454] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 0.
I20260923 20:22:26.147477 22810 leader_election.cc:302] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c3fd82504946a1dc1d006d73be17, 171094e880fc4392b44cb74916ddded2; no voters:
I20260923 20:22:26.147745 22958 raft_consensus.cc:2781] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:26.147855 22958 raft_consensus.cc:482] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:26.147965 22958 raft_consensus.cc:3037] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.149981 22958 raft_consensus.cc:504] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.150563 22958 leader_election.cc:288] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 election: Requested vote from peers a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177), 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311)
I20260923 20:22:26.151067 22604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d9bccfd0574581b2bc49e0e7850514" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956441541e9437e8902b2b0239c7734"
I20260923 20:22:26.151125 22743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d9bccfd0574581b2bc49e0e7850514" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c3fd82504946a1dc1d006d73be17"
I20260923 20:22:26.151309 22604 raft_consensus.cc:3037] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.151358 22743 raft_consensus.cc:3037] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.153275 22604 raft_consensus.cc:2454] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 1.
I20260923 20:22:26.153677 22809 leader_election.cc:302] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171094e880fc4392b44cb74916ddded2, a956441541e9437e8902b2b0239c7734; no voters:
I20260923 20:22:26.153956 22958 raft_consensus.cc:2781] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:26.154170 22958 raft_consensus.cc:686] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [term 1 LEADER]: Becoming Leader. State: Replica: 171094e880fc4392b44cb74916ddded2, State: Running, Role: LEADER
I20260923 20:22:26.154412 22958 consensus_queue.cc:237] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.154671 22743 raft_consensus.cc:2454] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 1.
I20260923 20:22:26.156250 22983 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: TxnStatusTablet state changed. Reason: New leader 171094e880fc4392b44cb74916ddded2. Latest consensus state: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:26.156579 22983 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:22:26.157246 22996 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:22:26.157481 22996 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:22:26.157783 22442 catalog_manager.cc:5495] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 171094e880fc4392b44cb74916ddded2 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } health_report { overall_health: HEALTHY } } }
I20260923 20:22:26.206799 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.199487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46703 (local address 127.0.0.1:43088)
0923 20:22:26.199689 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:26.199715 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:26.199883 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:26.201105 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:26.201113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:26.201131 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:22:26.201294 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.201302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.202633 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.202642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.204269 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.204286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.205019 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.205025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.205220 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.205299 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.206185 (+ 886us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.206207 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:22:26.206303 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 20:22:26.206469 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 20:22:26.207113 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.199552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.14.127.195:46703)
0923 20:22:26.200689 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.200693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.200712 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:22:26.200784 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.200788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.200859 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.201428 (+ 569us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:26.201632 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.202461 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.204466 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.204864 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.204955 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.205520 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.206067 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.206178 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.206532 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.206592 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":846,"threads_started":1}
I20260923 20:22:26.211054 22604 raft_consensus.cc:1261] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Refusing update from remote peer 171094e880fc4392b44cb74916ddded2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:26.211251 22743 raft_consensus.cc:1261] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Refusing update from remote peer 171094e880fc4392b44cb74916ddded2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:26.211902 22983 consensus_queue.cc:1035] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:26.212503 22958 consensus_queue.cc:1035] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:26.216861 22970 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: TxnStatusTablet state changed. Reason: New leader 171094e880fc4392b44cb74916ddded2. Latest consensus state: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:26.217177 22970 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:22:26.219341 22983 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:26.219602 22983 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:22:26.219785 22966 mvcc.cc:205] Tried to move back new op lower bound from 7332638499673088000 to 7332638499451441152. Current Snapshot: MvccSnapshot[applied={T|T < 7332638499673088000}]
I20260923 20:22:26.220639 22959 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: TxnStatusTablet state changed. Reason: New leader 171094e880fc4392b44cb74916ddded2. Latest consensus state: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:26.220923 22959 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:22:26.222788 22970 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:26.222977 22970 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P 12e8c3fd82504946a1dc1d006d73be17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:22:26.227130 22958 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:26.227463 22958 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P 171094e880fc4392b44cb74916ddded2: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:22:26.233120 22959 tablet_replica.cc:439] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } } }
I20260923 20:22:26.233341 22959 tablet_replica.cc:442] T 35d9bccfd0574581b2bc49e0e7850514 P a956441541e9437e8902b2b0239c7734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:22:26.248539 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.229915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.14.127.195:46703)
0923 20:22:26.230118 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.230126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.243601 (+ 13475us) server_negotiation.cc:407] Connection header received
0923 20:22:26.243825 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.243830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.243914 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.244047 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:26.244552 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.245320 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.246737 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.247174 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.247329 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.247739 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.247886 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.247981 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.248140 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.248218 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:22:26.268405 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.260228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:37187 (local address 127.14.127.195:46703)
0923 20:22:26.260479 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.260487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.263264 (+ 2777us) server_negotiation.cc:407] Connection header received
0923 20:22:26.263465 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.263470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.263527 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.263610 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:26.264181 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.265059 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.266597 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.267035 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.267191 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.267642 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.267727 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.267823 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.267962 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.268025 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:22:26.268452 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.260198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46703 (local address 127.14.127.195:37187)
0923 20:22:26.263151 (+ 2953us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:26.263181 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:26.263352 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:26.263628 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:26.263633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:26.263642 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:22:26.264008 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.264018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.265180 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.265188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.266461 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.266473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.267321 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.267325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.267417 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.267461 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.267828 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.267840 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:22:26.267932 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:22:26.268056 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2786,"thread_start_us":2648,"threads_started":1}
I20260923 20:22:26.279871 22743 raft_consensus.cc:1261] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Refusing update from remote peer 171094e880fc4392b44cb74916ddded2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:26.280107 22604 raft_consensus.cc:1261] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Refusing update from remote peer 171094e880fc4392b44cb74916ddded2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:26.280689 22983 consensus_queue.cc:1035] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:26.281015 23002 consensus_queue.cc:1035] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:26.296218 22967 mvcc.cc:205] Tried to move back new op lower bound from 7332638499957067776 to 7332638498306883584. Current Snapshot: MvccSnapshot[applied={T|T < 7332638499957067776}]
I20260923 20:22:26.360188 22441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:26.360481 22441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:26.363587 22441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: Sending DeleteTablet for 3 replicas of tablet a05e2ba2d0fd495f806919083f23d1f2
I20260923 20:22:26.364364 22579 tablet_service.cc:1514] Processing DeleteTablet for tablet a05e2ba2d0fd495f806919083f23d1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:48304
I20260923 20:22:26.364471 22718 tablet_service.cc:1514] Processing DeleteTablet for tablet a05e2ba2d0fd495f806919083f23d1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57734
I20260923 20:22:26.365222 22857 tablet_service.cc:1514] Processing DeleteTablet for tablet a05e2ba2d0fd495f806919083f23d1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43086
I20260923 20:22:26.380213 23011 tablet_replica.cc:330] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: stopping tablet replica
I20260923 20:22:26.380489 23011 raft_consensus.cc:2229] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:26.380888 23011 raft_consensus.cc:2258] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.382069 23011 ts_tablet_manager.cc:1905] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.389094 23012 tablet_replica.cc:330] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: stopping tablet replica
I20260923 20:22:26.389849 23013 tablet_replica.cc:330] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: stopping tablet replica
I20260923 20:22:26.390161 23012 raft_consensus.cc:2229] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:26.390377 23011 ts_tablet_manager.cc:1918] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:22:26.390507 23011 log.cc:1196] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/a05e2ba2d0fd495f806919083f23d1f2
I20260923 20:22:26.390511 23013 raft_consensus.cc:2229] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:26.390926 23012 raft_consensus.cc:2258] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.391010 23011 ts_tablet_manager.cc:1939] T a05e2ba2d0fd495f806919083f23d1f2 P a956441541e9437e8902b2b0239c7734: Deleting consensus metadata
I20260923 20:22:26.391773 23013 raft_consensus.cc:2258] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.392158 22427 catalog_manager.cc:4841] TS a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177): tablet a05e2ba2d0fd495f806919083f23d1f2 (table test-table [id=731cb18d09224f57afcc8bff16ca7a88]) successfully deleted
I20260923 20:22:26.393021 23012 ts_tablet_manager.cc:1905] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.395337 23013 ts_tablet_manager.cc:1905] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.399353 23012 ts_tablet_manager.cc:1918] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:22:26.399494 23012 log.cc:1196] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/a05e2ba2d0fd495f806919083f23d1f2
I20260923 20:22:26.400218 23012 ts_tablet_manager.cc:1939] T a05e2ba2d0fd495f806919083f23d1f2 P 12e8c3fd82504946a1dc1d006d73be17: Deleting consensus metadata
I20260923 20:22:26.401296 22428 catalog_manager.cc:4841] TS 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311): tablet a05e2ba2d0fd495f806919083f23d1f2 (table test-table [id=731cb18d09224f57afcc8bff16ca7a88]) successfully deleted
I20260923 20:22:26.402786 23013 ts_tablet_manager.cc:1918] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:22:26.402946 23013 log.cc:1196] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/a05e2ba2d0fd495f806919083f23d1f2
I20260923 20:22:26.403386 23013 ts_tablet_manager.cc:1939] T a05e2ba2d0fd495f806919083f23d1f2 P 171094e880fc4392b44cb74916ddded2: Deleting consensus metadata
I20260923 20:22:26.404246 22428 catalog_manager.cc:4841] TS 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703): tablet a05e2ba2d0fd495f806919083f23d1f2 (table test-table [id=731cb18d09224f57afcc8bff16ca7a88]) successfully deleted
Sep 23 20:22:26 dist-test-slave-wmd0 krb5kdc[22397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:22:26 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:26 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:22:26.436535 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:26.425231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.14.127.254:46083)
0923 20:22:26.425496 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.425502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.425899 (+ 397us) server_negotiation.cc:407] Connection header received
0923 20:22:26.426079 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.426083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.426146 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.426231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:26.427997 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.428819 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.430252 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.430826 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.430971 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.436214 (+ 5243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58210: BlockingRecv error: recv got EOF from 127.0.0.1:58210 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 20:22:26 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM
I20260923 20:22:26.455262 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.439587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.14.127.254:46083)
0923 20:22:26.439816 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.439823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.440115 (+ 292us) server_negotiation.cc:407] Connection header received
0923 20:22:26.440256 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.440260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.440330 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.440414 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:26.441880 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.442737 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.444491 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.444885 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.444990 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.449234 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:26.449240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:26.449246 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:26.449289 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:26.452691 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:26.453323 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:26.453330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:26.453334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:26.453506 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:26.453888 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:26.453896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:26.453900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:26.454223 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:26.454469 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.454707 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.454938 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:22:26.458927 22441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:26.459570 22441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:26.466864 22718 tablet_service.cc:1467] Processing CreateTablet for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 (DEFAULT_TABLE table=test-table [id=6c9b60cb979648648d645e038e99c00b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:26.466961 22579 tablet_service.cc:1467] Processing CreateTablet for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 (DEFAULT_TABLE table=test-table [id=6c9b60cb979648648d645e038e99c00b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:26.467326 22718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:26.467422 22579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:26.467373 22857 tablet_service.cc:1467] Processing CreateTablet for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 (DEFAULT_TABLE table=test-table [id=6c9b60cb979648648d645e038e99c00b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:26.467788 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:26.479862 23026 tablet_bootstrap.cc:492] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: Bootstrap starting.
I20260923 20:22:26.482079 23028 tablet_bootstrap.cc:492] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: Bootstrap starting.
I20260923 20:22:26.482455 23027 tablet_bootstrap.cc:492] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: Bootstrap starting.
I20260923 20:22:26.483016 23026 tablet_bootstrap.cc:654] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:26.484104 23028 tablet_bootstrap.cc:654] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:26.484954 23027 tablet_bootstrap.cc:654] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:26.489066 23027 tablet_bootstrap.cc:492] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: No bootstrap required, opened a new log
I20260923 20:22:26.489234 23027 ts_tablet_manager.cc:1397] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260923 20:22:26.490178 23027 raft_consensus.cc:348] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.490513 23027 raft_consensus.cc:374] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:26.490597 23027 raft_consensus.cc:729] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a956441541e9437e8902b2b0239c7734, State: Initialized, Role: FOLLOWER
I20260923 20:22:26.490947 23027 consensus_queue.cc:260] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.494825 23027 ts_tablet_manager.cc:1428] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260923 20:22:26.498853 23026 tablet_bootstrap.cc:492] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: No bootstrap required, opened a new log
I20260923 20:22:26.499114 23026 ts_tablet_manager.cc:1397] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20260923 20:22:26.499832 23028 tablet_bootstrap.cc:492] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: No bootstrap required, opened a new log
I20260923 20:22:26.499992 23028 ts_tablet_manager.cc:1397] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.009s
I20260923 20:22:26.499980 23026 raft_consensus.cc:348] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.500255 23026 raft_consensus.cc:374] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:26.500355 23026 raft_consensus.cc:729] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c3fd82504946a1dc1d006d73be17, State: Initialized, Role: FOLLOWER
I20260923 20:22:26.500614 23026 consensus_queue.cc:260] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.500908 23028 raft_consensus.cc:348] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.501164 23026 ts_tablet_manager.cc:1428] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:22:26.501180 23028 raft_consensus.cc:374] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:26.501303 23028 raft_consensus.cc:729] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171094e880fc4392b44cb74916ddded2, State: Initialized, Role: FOLLOWER
I20260923 20:22:26.501613 23028 consensus_queue.cc:260] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.502368 23028 ts_tablet_manager.cc:1428] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 20:22:26.581997 22983 raft_consensus.cc:482] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:26.582230 22983 raft_consensus.cc:504] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.583014 22983 leader_election.cc:288] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177), 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311)
I20260923 20:22:26.583563 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ef0e73f7fa42c4ace4c5f611f9ec44" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956441541e9437e8902b2b0239c7734" is_pre_election: true
I20260923 20:22:26.583684 22742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ef0e73f7fa42c4ace4c5f611f9ec44" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c3fd82504946a1dc1d006d73be17" is_pre_election: true
I20260923 20:22:26.583868 22603 raft_consensus.cc:2454] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 0.
I20260923 20:22:26.583935 22742 raft_consensus.cc:2454] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 0.
I20260923 20:22:26.584379 22809 leader_election.cc:302] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171094e880fc4392b44cb74916ddded2, a956441541e9437e8902b2b0239c7734; no voters:
I20260923 20:22:26.584642 22983 raft_consensus.cc:2781] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:26.584801 22983 raft_consensus.cc:482] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:26.584923 22983 raft_consensus.cc:3037] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.587283 22983 raft_consensus.cc:504] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.587889 22983 leader_election.cc:288] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 election: Requested vote from peers a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177), 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311)
I20260923 20:22:26.588367 22742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ef0e73f7fa42c4ace4c5f611f9ec44" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c3fd82504946a1dc1d006d73be17"
I20260923 20:22:26.588408 22603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ef0e73f7fa42c4ace4c5f611f9ec44" candidate_uuid: "171094e880fc4392b44cb74916ddded2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a956441541e9437e8902b2b0239c7734"
I20260923 20:22:26.588567 22742 raft_consensus.cc:3037] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.588666 22603 raft_consensus.cc:3037] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.590833 22742 raft_consensus.cc:2454] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 1.
I20260923 20:22:26.591269 22810 leader_election.cc:302] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c3fd82504946a1dc1d006d73be17, 171094e880fc4392b44cb74916ddded2; no voters:
I20260923 20:22:26.591542 22983 raft_consensus.cc:2781] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:26.591710 22983 raft_consensus.cc:686] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 1 LEADER]: Becoming Leader. State: Replica: 171094e880fc4392b44cb74916ddded2, State: Running, Role: LEADER
I20260923 20:22:26.592084 22983 consensus_queue.cc:237] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.592408 22603 raft_consensus.cc:2454] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171094e880fc4392b44cb74916ddded2 in term 1.
I20260923 20:22:26.595220 22442 catalog_manager.cc:5495] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 171094e880fc4392b44cb74916ddded2 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "171094e880fc4392b44cb74916ddded2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } health_report { overall_health: HEALTHY } } }
I20260923 20:22:26.638365 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.633229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.14.127.195:46703)
0923 20:22:26.633423 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.633429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.633449 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:22:26.633635 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.633641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.633778 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.633885 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:26.634277 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.635096 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.636516 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.637015 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.637145 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.637607 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.637747 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.637850 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.637990 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.638069 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:22:26.641098 22742 raft_consensus.cc:1261] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Refusing update from remote peer 171094e880fc4392b44cb74916ddded2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:26.641170 22603 raft_consensus.cc:1261] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Refusing update from remote peer 171094e880fc4392b44cb74916ddded2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:26.641697 22983 consensus_queue.cc:1035] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:26.641957 22958 consensus_queue.cc:1035] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:26.660020 22442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:26.660382 22442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:26.665443 22442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: Sending DeleteTablet for 3 replicas of tablet 91ef0e73f7fa42c4ace4c5f611f9ec44
I20260923 20:22:26.666651 22718 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57734
I20260923 20:22:26.666648 22579 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:48304
I20260923 20:22:26.666920 23011 tablet_replica.cc:330] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: stopping tablet replica
I20260923 20:22:26.666909 23012 tablet_replica.cc:330] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: stopping tablet replica
I20260923 20:22:26.667102 23012 raft_consensus.cc:2229] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:26.667098 23011 raft_consensus.cc:2229] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:26.667320 23012 raft_consensus.cc:2258] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.667320 23011 raft_consensus.cc:2258] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.668331 23011 ts_tablet_manager.cc:1905] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.668445 22857 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43086
I20260923 20:22:26.668691 23013 tablet_replica.cc:330] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: stopping tablet replica
I20260923 20:22:26.668877 23013 raft_consensus.cc:2229] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:26.669188 23013 raft_consensus.cc:2258] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.670118 23012 ts_tablet_manager.cc:1905] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.670775 22857 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43086
I20260923 20:22:26.671729 22718 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57734
I20260923 20:22:26.672731 23013 ts_tablet_manager.cc:1905] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.675884 22442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:26.676532 22442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:26.677068 23011 ts_tablet_manager.cc:1918] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:26.677228 23011 log.cc:1196] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/91ef0e73f7fa42c4ace4c5f611f9ec44
I20260923 20:22:26.677215 23012 ts_tablet_manager.cc:1918] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:26.677366 23012 log.cc:1196] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/91ef0e73f7fa42c4ace4c5f611f9ec44
I20260923 20:22:26.677663 23011 ts_tablet_manager.cc:1939] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P a956441541e9437e8902b2b0239c7734: Deleting consensus metadata
I20260923 20:22:26.677821 23012 ts_tablet_manager.cc:1939] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 12e8c3fd82504946a1dc1d006d73be17: Deleting consensus metadata
I20260923 20:22:26.679682 22427 catalog_manager.cc:4841] TS a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177): tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 (table test-table [id=6c9b60cb979648648d645e038e99c00b]) successfully deleted
I20260923 20:22:26.679796 22428 catalog_manager.cc:4841] TS 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311): tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 (table test-table [id=6c9b60cb979648648d645e038e99c00b]) successfully deleted
W20260923 20:22:26.680256 22428 catalog_manager.cc:4808] TS 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311): delete failed for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 because the tablet was not found. No further retry: Not found: Tablet not found: 91ef0e73f7fa42c4ace4c5f611f9ec44
I20260923 20:22:26.683058 23013 ts_tablet_manager.cc:1918] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:26.683223 23013 log.cc:1196] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/91ef0e73f7fa42c4ace4c5f611f9ec44
I20260923 20:22:26.683640 23013 ts_tablet_manager.cc:1939] T 91ef0e73f7fa42c4ace4c5f611f9ec44 P 171094e880fc4392b44cb74916ddded2: Deleting consensus metadata
I20260923 20:22:26.684532 22579 tablet_service.cc:1467] Processing CreateTablet for tablet a360482df7594f26b41d8622697807b9 (DEFAULT_TABLE table=test-table [id=d3272f4acd944d6aaea8fdef637436db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:26.684989 22579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a360482df7594f26b41d8622697807b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:26.685225 22428 catalog_manager.cc:4841] TS 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703): tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 (table test-table [id=6c9b60cb979648648d645e038e99c00b]) successfully deleted
W20260923 20:22:26.685456 22428 catalog_manager.cc:4808] TS 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703): delete failed for tablet 91ef0e73f7fa42c4ace4c5f611f9ec44 because the tablet was not found. No further retry: Not found: Tablet not found: 91ef0e73f7fa42c4ace4c5f611f9ec44
I20260923 20:22:26.685633 22718 tablet_service.cc:1467] Processing CreateTablet for tablet a360482df7594f26b41d8622697807b9 (DEFAULT_TABLE table=test-table [id=d3272f4acd944d6aaea8fdef637436db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:26.685987 22718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a360482df7594f26b41d8622697807b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:26.687294 22857 tablet_service.cc:1467] Processing CreateTablet for tablet a360482df7594f26b41d8622697807b9 (DEFAULT_TABLE table=test-table [id=d3272f4acd944d6aaea8fdef637436db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:26.687675 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a360482df7594f26b41d8622697807b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:26.692296 23028 tablet_bootstrap.cc:492] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: Bootstrap starting.
I20260923 20:22:26.692934 23026 tablet_bootstrap.cc:492] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: Bootstrap starting.
I20260923 20:22:26.693957 23028 tablet_bootstrap.cc:654] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:26.694650 23026 tablet_bootstrap.cc:654] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:26.697849 23027 tablet_bootstrap.cc:492] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: Bootstrap starting.
I20260923 20:22:26.699966 23027 tablet_bootstrap.cc:654] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:26.703006 23027 tablet_bootstrap.cc:492] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: No bootstrap required, opened a new log
I20260923 20:22:26.703195 23027 ts_tablet_manager.cc:1397] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260923 20:22:26.703807 23027 raft_consensus.cc:348] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.704039 23027 raft_consensus.cc:374] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:26.704116 23027 raft_consensus.cc:729] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a956441541e9437e8902b2b0239c7734, State: Initialized, Role: FOLLOWER
I20260923 20:22:26.704401 23027 consensus_queue.cc:260] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.705668 23027 ts_tablet_manager.cc:1428] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:22:26.708225 23026 tablet_bootstrap.cc:492] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: No bootstrap required, opened a new log
I20260923 20:22:26.708415 23026 ts_tablet_manager.cc:1397] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260923 20:22:26.708431 23028 tablet_bootstrap.cc:492] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: No bootstrap required, opened a new log
I20260923 20:22:26.708592 23028 ts_tablet_manager.cc:1397] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260923 20:22:26.709161 23028 raft_consensus.cc:348] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.709339 23028 raft_consensus.cc:374] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:26.709398 23028 raft_consensus.cc:729] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171094e880fc4392b44cb74916ddded2, State: Initialized, Role: FOLLOWER
I20260923 20:22:26.709227 23026 raft_consensus.cc:348] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.709491 23026 raft_consensus.cc:374] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:26.709579 23026 raft_consensus.cc:729] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c3fd82504946a1dc1d006d73be17, State: Initialized, Role: FOLLOWER
I20260923 20:22:26.709626 23028 consensus_queue.cc:260] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.709839 23026 consensus_queue.cc:260] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.710170 23028 ts_tablet_manager.cc:1428] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 20:22:26.710412 23026 ts_tablet_manager.cc:1428] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 20:22:26.760761 22959 raft_consensus.cc:482] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:26.761031 22959 raft_consensus.cc:504] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.762974 22959 leader_election.cc:288] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311), 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703)
I20260923 20:22:26.771977 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.763150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:37311 (local address 127.14.127.193:50359)
0923 20:22:26.763752 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:26.763776 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:26.763919 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:26.764956 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:26.764961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:26.764984 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:26.765394 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.765410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.766897 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.766905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.769050 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.769064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.771177 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.771184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.771365 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.771403 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:22:26.771467 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 20:22:26.771600 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":1,"thread_start_us":304,"threads_started":1}
I20260923 20:22:26.772954 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.763249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:50359 (local address 127.14.127.194:37311)
0923 20:22:26.764596 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.764603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.764624 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:22:26.764701 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.764707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.764788 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.764898 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:26.765596 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.766715 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.769628 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.770972 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.772368 (+ 1396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.772436 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.772485 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.772624 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":1051,"threads_started":1}
I20260923 20:22:26.773473 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.764367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:49403 (local address 127.14.127.195:46703)
0923 20:22:26.764561 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.764567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.765204 (+ 637us) server_negotiation.cc:407] Connection header received
0923 20:22:26.765284 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.765290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.765373 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.765501 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:26.766134 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.767475 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.770356 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.771917 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.772069 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.772147 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.773014 (+ 867us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.773165 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:22:26.773465 22742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a360482df7594f26b41d8622697807b9" candidate_uuid: "a956441541e9437e8902b2b0239c7734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c3fd82504946a1dc1d006d73be17" is_pre_election: true
I20260923 20:22:26.773729 22742 raft_consensus.cc:2454] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a956441541e9437e8902b2b0239c7734 in term 0.
I20260923 20:22:26.774241 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.763967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46703 (local address 127.14.127.193:49403)
0923 20:22:26.764802 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:26.764816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:26.764956 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:26.765522 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:26.765528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:26.765559 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:26.765929 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.765941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.767721 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.767731 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.770156 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.770172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.772600 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.772607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.772810 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.772847 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:22:26.773720 (+ 873us) client_negotiation.cc:240] Negotiation successful
0923 20:22:26.773868 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":590,"threads_started":1}
I20260923 20:22:26.774273 22531 leader_election.cc:302] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c3fd82504946a1dc1d006d73be17, a956441541e9437e8902b2b0239c7734; no voters:
I20260923 20:22:26.774786 22959 raft_consensus.cc:2781] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:26.774912 22959 raft_consensus.cc:482] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:26.775008 22959 raft_consensus.cc:3037] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.775228 22880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a360482df7594f26b41d8622697807b9" candidate_uuid: "a956441541e9437e8902b2b0239c7734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171094e880fc4392b44cb74916ddded2" is_pre_election: true
I20260923 20:22:26.775496 22880 raft_consensus.cc:2454] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a956441541e9437e8902b2b0239c7734 in term 0.
I20260923 20:22:26.778456 22959 raft_consensus.cc:504] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.779189 22959 leader_election.cc:288] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311), 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703)
I20260923 20:22:26.780350 22742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a360482df7594f26b41d8622697807b9" candidate_uuid: "a956441541e9437e8902b2b0239c7734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c3fd82504946a1dc1d006d73be17"
I20260923 20:22:26.780581 22742 raft_consensus.cc:3037] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.783973 22742 raft_consensus.cc:2454] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a956441541e9437e8902b2b0239c7734 in term 1.
I20260923 20:22:26.784472 22531 leader_election.cc:302] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c3fd82504946a1dc1d006d73be17, a956441541e9437e8902b2b0239c7734; no voters:
I20260923 20:22:26.784543 22880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a360482df7594f26b41d8622697807b9" candidate_uuid: "a956441541e9437e8902b2b0239c7734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171094e880fc4392b44cb74916ddded2"
I20260923 20:22:26.784747 22959 raft_consensus.cc:2781] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:26.784768 22880 raft_consensus.cc:3037] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:26.788244 22880 raft_consensus.cc:2454] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a956441541e9437e8902b2b0239c7734 in term 1.
I20260923 20:22:26.789657 22959 raft_consensus.cc:686] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 1 LEADER]: Becoming Leader. State: Replica: a956441541e9437e8902b2b0239c7734, State: Running, Role: LEADER
I20260923 20:22:26.790218 22959 consensus_queue.cc:237] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } }
I20260923 20:22:26.794065 22440 catalog_manager.cc:5495] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 reported cstate change: term changed from 0 to 1, leader changed from <none> to a956441541e9437e8902b2b0239c7734 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "a956441541e9437e8902b2b0239c7734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a956441541e9437e8902b2b0239c7734" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 42177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 } health_report { overall_health: UNKNOWN } } }
I20260923 20:22:26.824329 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.818443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.14.127.193:42177)
0923 20:22:26.819102 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.819106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.819123 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:22:26.819199 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.819202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.819275 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.819452 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:26.819863 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.820635 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.821917 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.822338 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.822475 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.823033 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.823679 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.823833 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.823927 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.824006 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":419,"threads_started":1}
I20260923 20:22:26.840560 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.835307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:33623 (local address 127.14.127.195:46703)
0923 20:22:26.835530 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.835538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.835733 (+ 195us) server_negotiation.cc:407] Connection header received
0923 20:22:26.835939 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.835948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.836017 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.836123 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:26.836600 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.837395 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.838808 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.839281 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.839429 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.839813 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.839912 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.840014 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.840149 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.840232 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:22:26.840631 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.835085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46703 (local address 127.14.127.193:33623)
0923 20:22:26.835635 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:26.835650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:26.835806 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:26.836137 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:26.836142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:26.836153 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:22:26.836445 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.836452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.837544 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.837555 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.838680 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.838692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.839454 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.839460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.839593 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.839640 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.840015 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.840028 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 20:22:26.840129 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:22:26.840266 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":457,"threads_started":1}
I20260923 20:22:26.849548 22742 raft_consensus.cc:1261] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Refusing update from remote peer a956441541e9437e8902b2b0239c7734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:26.849545 22880 raft_consensus.cc:1261] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Refusing update from remote peer a956441541e9437e8902b2b0239c7734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:26.850214 23042 consensus_queue.cc:1035] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171094e880fc4392b44cb74916ddded2" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:26.850777 22959 consensus_queue.cc:1035] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c3fd82504946a1dc1d006d73be17" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:26.886086 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.875644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:39027 (local address 127.14.127.193:42177)
0923 20:22:26.875968 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.875974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.879252 (+ 3278us) server_negotiation.cc:407] Connection header received
0923 20:22:26.879437 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.879441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.879500 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.879570 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:26.880768 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.881939 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.884238 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.884773 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.884918 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.885368 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.885509 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.885605 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.885827 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.885897 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260923 20:22:26.886415 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.875347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:42177 (local address 127.14.127.195:39027)
0923 20:22:26.879139 (+ 3792us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:26.879165 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:26.879314 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:26.879622 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:26.879632 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:26.879646 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:22:26.879980 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.880168 (+ 188us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.882081 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.882087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.884077 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:26.884093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.884910 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.884916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:26.885109 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.885155 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:22:26.885665 (+ 510us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:22:26.885678 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 20:22:26.885801 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 20:22:26.885963 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3614,"thread_start_us":3474,"threads_started":1}
I20260923 20:22:26.923070 22440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:26.923373 22440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:26.927121 22440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32e2d393000c403880a2519dcc9ffdc0: Sending DeleteTablet for 3 replicas of tablet a360482df7594f26b41d8622697807b9
I20260923 20:22:26.927920 22579 tablet_service.cc:1514] Processing DeleteTablet for tablet a360482df7594f26b41d8622697807b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:48304
I20260923 20:22:26.928323 23011 tablet_replica.cc:330] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: stopping tablet replica
I20260923 20:22:26.928524 23011 raft_consensus.cc:2229] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:26.929032 23011 raft_consensus.cc:2258] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.929359 22718 tablet_service.cc:1514] Processing DeleteTablet for tablet a360482df7594f26b41d8622697807b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57734
I20260923 20:22:26.929649 23012 tablet_replica.cc:330] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: stopping tablet replica
I20260923 20:22:26.929821 23012 raft_consensus.cc:2229] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:26.930025 23012 raft_consensus.cc:2258] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.931283 23012 ts_tablet_manager.cc:1905] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.931521 23011 ts_tablet_manager.cc:1905] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.935118 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 22513
I20260923 20:22:26.938465 22857 tablet_service.cc:1514] Processing DeleteTablet for tablet a360482df7594f26b41d8622697807b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:26 UTC) from {username='oryx', principal='oryx/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:43086
I20260923 20:22:26.938591 23012 ts_tablet_manager.cc:1918] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:22:26.938764 23012 log.cc:1196] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/a360482df7594f26b41d8622697807b9
I20260923 20:22:26.938789 23013 tablet_replica.cc:330] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: stopping tablet replica
I20260923 20:22:26.938970 23013 raft_consensus.cc:2229] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:26.939174 23013 raft_consensus.cc:2258] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:26.939200 23012 ts_tablet_manager.cc:1939] T a360482df7594f26b41d8622697807b9 P 12e8c3fd82504946a1dc1d006d73be17: Deleting consensus metadata
I20260923 20:22:26.940372 22428 catalog_manager.cc:4841] TS 12e8c3fd82504946a1dc1d006d73be17 (127.14.127.194:37311): tablet a360482df7594f26b41d8622697807b9 (table test-table [id=d3272f4acd944d6aaea8fdef637436db]) successfully deleted
I20260923 20:22:26.941332 23013 ts_tablet_manager.cc:1905] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:26.951428 23013 ts_tablet_manager.cc:1918] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:22:26.951586 23013 log.cc:1196] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/a360482df7594f26b41d8622697807b9
I20260923 20:22:26.952054 23013 ts_tablet_manager.cc:1939] T a360482df7594f26b41d8622697807b9 P 171094e880fc4392b44cb74916ddded2: Deleting consensus metadata
I20260923 20:22:26.952245 23011 ts_tablet_manager.cc:1918] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:22:26.952459 23011 log.cc:1196] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/a360482df7594f26b41d8622697807b9
I20260923 20:22:26.952982 23011 ts_tablet_manager.cc:1939] T a360482df7594f26b41d8622697807b9 P a956441541e9437e8902b2b0239c7734: Deleting consensus metadata
I20260923 20:22:26.954250 22427 catalog_manager.cc:4841] TS a956441541e9437e8902b2b0239c7734 (127.14.127.193:42177): tablet a360482df7594f26b41d8622697807b9 (table test-table [id=d3272f4acd944d6aaea8fdef637436db]) successfully deleted
I20260923 20:22:26.955597 22428 catalog_manager.cc:4841] TS 171094e880fc4392b44cb74916ddded2 (127.14.127.195:46703): tablet a360482df7594f26b41d8622697807b9 (table test-table [id=d3272f4acd944d6aaea8fdef637436db]) successfully deleted
Sep 23 20:22:26 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.193@KRBTEST.COM
I20260923 20:22:26.975132 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:26.936850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.14.127.193:42177)
0923 20:22:26.937326 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:26.937333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:26.955086 (+ 17753us) server_negotiation.cc:407] Connection header received
0923 20:22:26.955376 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:26.955384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:26.955499 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:26.955582 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:26.957460 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.958600 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.960572 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:26.961361 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:26.961490 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:26.967082 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:26.967098 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:26.967120 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:26.967164 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:26.971824 (+ 4660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:26.972654 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:26.972666 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:26.972671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:26.972858 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:26.973281 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:26.973286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:26.973290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:26.973749 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:26.974105 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:26.974359 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 20:22:26.974666 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
I20260923 20:22:26.976212 22644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:27.086931 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 22513
I20260923 20:22:27.107530 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 22653
Sep 23 20:22:27 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.194@KRBTEST.COM
I20260923 20:22:27.122831 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:27.108348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.14.127.194:37311)
0923 20:22:27.108646 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:27.108652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:27.108673 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:22:27.108741 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:27.108746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:27.108820 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:27.108934 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:27.110099 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:27.110844 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:27.112040 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:27.112421 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:27.112552 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:27.117012 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:27.117026 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:27.117041 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:27.117079 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:27.120017 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:27.120685 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:27.120692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:27.120695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:27.120856 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:27.121213 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:27.121218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:27.121222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:27.121580 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:27.121885 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:27.122200 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 20:22:27.122473 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260923 20:22:27.123591 22783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:27.245394 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 22653
I20260923 20:22:27.267206 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 22792
Sep 23 20:22:27 dist-test-slave-wmd0 krb5kdc[22397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.127.195@KRBTEST.COM
I20260923 20:22:27.282927 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:27.268016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.14.127.195:46703)
0923 20:22:27.268182 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:27.268186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:27.268205 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:22:27.268325 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:27.268329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:27.268421 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:27.268508 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:27.269753 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:27.270573 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:27.272030 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:27.272450 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:27.272549 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:27.276870 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:27.276885 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:27.276902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:27.276936 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:27.279943 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:27.280643 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:27.280652 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:27.280656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:27.280817 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:27.281329 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:27.281334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:27.281338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:27.281825 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:27.282126 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:27.282339 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 20:22:27.282556 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:22:27.284013 22922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:27.427963 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 22792
I20260923 20:22:27.448653 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 22410
I20260923 20:22:27.449793 22482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:27.563898 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 22410
2026-09-23T20:22:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5679 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:22:27 dist-test-slave-wmd0 krb5kdc[23065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:22:27 dist-test-slave-wmd0 krb5kdc[23065](info): set up 2 sockets
Sep 23 20:22:27 dist-test-slave-wmd0 krb5kdc[23065](info): commencing operation
krb5kdc: starting...
W20260923 20:22:28.692042 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:22:28 dist-test-slave-wmd0 krb5kdc[23065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:22:28.906816 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46095
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:46095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:29.226634 23078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:29.226898 23078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:29.227080 23078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:29.234192 23078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:29.234308 23078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:29.234396 23078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:29.234483 23078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:29.234562 23078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:29.244511 23078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:46095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:29.245834 23078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:29.247153 23078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:29.258591 23083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:29.258665 23086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:29.258699 23084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:29.259881 23078 server_base.cc:1034] running on GCE node
I20260923 20:22:29.260651 23078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:29.262064 23078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:29.263306 23078 hybrid_clock.cc:648] HybridClock initialized: now 1790194949263265 us; error 63 us; skew 500 ppm
I20260923 20:22:29.263644 23078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T20:22:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1768 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:22:29 dist-test-slave-wmd0 krb5kdc[23092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:22:29 dist-test-slave-wmd0 krb5kdc[23092](info): set up 2 sockets
Sep 23 20:22:29 dist-test-slave-wmd0 krb5kdc[23092](info): commencing operation
krb5kdc: starting...
W20260923 20:22:30.435976 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:22:30 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:22:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:22:30.650911 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36301
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:36301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:30.970234 23105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:30.970532 23105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:30.970667 23105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:30.977916 23105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:30.978022 23105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:30.978067 23105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:30.978153 23105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:30.978220 23105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:30.988147 23105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:36301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:30.989815 23105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:30.991232 23105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:31.002930 23113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:31.002990 23110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:31.002956 23111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:31.004033 23105 server_base.cc:1034] running on GCE node
I20260923 20:22:31.004828 23105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:31.006102 23105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:31.007339 23105 hybrid_clock.cc:648] HybridClock initialized: now 1790194951007269 us; error 32 us; skew 500 ppm
I20260923 20:22:31.007635 23105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:31 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:31.011076 23105 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:22:31.012598 23105 webserver.cc:460] Webserver started at http://127.14.127.254:35677/ using document root <none> and password file <none>
I20260923 20:22:31.013232 23105 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:31.013329 23105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:31.013581 23105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:31.016496 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "389cc8e2af9f4b889d513dfc34a7b59b"
format_stamp: "Formatted at 2026-09-23 20:22:31 on dist-test-slave-wmd0"
server_key: "4e3fbc00b9452228e8f1e1d2ca685e3d"
server_key_iv: "b1137907d832c372f50278c2c2180978"
server_key_version: "encryptionkey@0"
I20260923 20:22:31.017194 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "389cc8e2af9f4b889d513dfc34a7b59b"
format_stamp: "Formatted at 2026-09-23 20:22:31 on dist-test-slave-wmd0"
server_key: "4e3fbc00b9452228e8f1e1d2ca685e3d"
server_key_iv: "b1137907d832c372f50278c2c2180978"
server_key_version: "encryptionkey@0"
I20260923 20:22:31.022168 23105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:22:31.025494 23120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:31.026847 23105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:22:31.027020 23105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "389cc8e2af9f4b889d513dfc34a7b59b"
format_stamp: "Formatted at 2026-09-23 20:22:31 on dist-test-slave-wmd0"
server_key: "4e3fbc00b9452228e8f1e1d2ca685e3d"
server_key_iv: "b1137907d832c372f50278c2c2180978"
server_key_version: "encryptionkey@0"
I20260923 20:22:31.027209 23105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:31.040252 23105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:31.044185 23105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:31.044468 23105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:31.062465 23105 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:36301
I20260923 20:22:31.062485 23171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:36301 every 8 connection(s)
I20260923 20:22:31.063819 23105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:31.067528 23172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:31.069945 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 23105
I20260923 20:22:31.070153 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:31.070843 14847 external_mini_cluster.cc:1428] Setting key 6415962a936f0802c2dbcbf8e0427417
I20260923 20:22:31.079516 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Bootstrap starting.
I20260923 20:22:31.083348 23172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:31.084668 23172 log.cc:824] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:22:31 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:31.087849 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: No bootstrap required, opened a new log
I20260923 20:22:31.093750 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:31.072741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.14.127.254:36301)
0923 20:22:31.073614 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:31.073625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:31.073664 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:22:31.075385 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:31.075406 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:31.075769 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:31.076081 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:31.077925 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:31.079252 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:31.080998 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:31.081682 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:31.081803 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:31.086802 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:31.086812 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:31.086825 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:31.086856 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:31.089844 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:31.090683 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:31.090689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:31.090698 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:31.090861 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:31.091242 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:31.091245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:31.091247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:31.091755 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:31.091954 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:31.092394 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:22:31.092770 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":386,"threads_started":1}
I20260923 20:22:31.096539 23172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } }
I20260923 20:22:31.096889 23172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:31.096997 23172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389cc8e2af9f4b889d513dfc34a7b59b, State: Initialized, Role: FOLLOWER
I20260923 20:22:31.097496 23172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } }
I20260923 20:22:31.097731 23172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:31.097877 23172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:31.098073 23172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:31.100304 23172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } }
I20260923 20:22:31.100768 23172 leader_election.cc:302] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 389cc8e2af9f4b889d513dfc34a7b59b; no voters:
I20260923 20:22:31.101432 23172 leader_election.cc:288] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:31.101559 23177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:31.102309 23177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 1 LEADER]: Becoming Leader. State: Replica: 389cc8e2af9f4b889d513dfc34a7b59b, State: Running, Role: LEADER
I20260923 20:22:31.102867 23177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } }
I20260923 20:22:31.103629 23172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:31.105458 23179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389cc8e2af9f4b889d513dfc34a7b59b. Latest consensus state: current_term: 1 leader_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } } }
I20260923 20:22:31.105844 23179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:31.107102 23178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } } }
I20260923 20:22:31.107409 23178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:31.108161 23184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:31.113329 23184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:31.122126 23184 catalog_manager.cc:1347] Generated new cluster ID: df0407aa9dfe43549e3074d8a62eac3b
I20260923 20:22:31.122257 23184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:31.146646 23184 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:31.147967 23184 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:31.153097 23184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Generated new TSK 0
I20260923 20:22:31.154134 23184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:22:31.218024 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36301
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:31.540742 23198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:31.541007 23198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:31.541172 23198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:31.548404 23198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:31.548511 23198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:31.548691 23198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:31.558095 23198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:31.559412 23198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:31.560758 23198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:31.573747 23206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:31.573783 23204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:31.573926 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:31.574374 23198 server_base.cc:1034] running on GCE node
I20260923 20:22:31.575060 23198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:31.575982 23198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:31.577211 23198 hybrid_clock.cc:648] HybridClock initialized: now 1790194951577173 us; error 58 us; skew 500 ppm
I20260923 20:22:31.577554 23198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:31 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:31.580732 23198 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:22:31.582086 23198 webserver.cc:460] Webserver started at http://127.14.127.193:34201/ using document root <none> and password file <none>
I20260923 20:22:31.582803 23198 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:31.582904 23198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:31.583148 23198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:31.585886 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "93bb1e6aa5e34f96925bd13c1ee8de75"
format_stamp: "Formatted at 2026-09-23 20:22:31 on dist-test-slave-wmd0"
server_key: "eeaa4b8d521460e85d66b173af771572"
server_key_iv: "01bb34485bea341b3d0360ab10a005a5"
server_key_version: "encryptionkey@0"
I20260923 20:22:31.586691 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "93bb1e6aa5e34f96925bd13c1ee8de75"
format_stamp: "Formatted at 2026-09-23 20:22:31 on dist-test-slave-wmd0"
server_key: "eeaa4b8d521460e85d66b173af771572"
server_key_iv: "01bb34485bea341b3d0360ab10a005a5"
server_key_version: "encryptionkey@0"
I20260923 20:22:31.591746 23198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:22:31.595082 23213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:31.596377 23198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:22:31.596544 23198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "93bb1e6aa5e34f96925bd13c1ee8de75"
format_stamp: "Formatted at 2026-09-23 20:22:31 on dist-test-slave-wmd0"
server_key: "eeaa4b8d521460e85d66b173af771572"
server_key_iv: "01bb34485bea341b3d0360ab10a005a5"
server_key_version: "encryptionkey@0"
I20260923 20:22:31.596724 23198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:31.612133 23198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:31.616091 23198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:31.616379 23198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:31.617321 23198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:31.618709 23198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:31.618781 23198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:31.618944 23198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:31.619006 23198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:31.653661 23198 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:46059
I20260923 20:22:31.653702 23325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:46059 every 8 connection(s)
I20260923 20:22:31.655045 23198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:31.660813 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 23198
I20260923 20:22:31.661021 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:31.661755 14847 external_mini_cluster.cc:1428] Setting key c48061a7783e4ac2774c9b59855d3f58
Sep 23 20:22:31 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194951, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:31.675755 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:31.657618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:57345 (local address 127.14.127.254:36301)
0923 20:22:31.658039 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:31.658045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:31.659172 (+ 1127us) server_negotiation.cc:407] Connection header received
0923 20:22:31.661038 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:31.661045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:31.661139 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:31.661271 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:31.663042 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:31.663912 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:31.665422 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:31.666164 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:31.666299 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:31.669970 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:31.669983 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:31.669986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:31.670017 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:31.672548 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:31.673489 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:31.673492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:31.673495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:31.673607 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:31.674081 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:31.674084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:31.674086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:31.674329 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:31.674433 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:31.675333 (+ 900us) server_negotiation.cc:299] Negotiation successful
0923 20:22:31.675498 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":300,"thread_start_us":199,"threads_started":1}
I20260923 20:22:31.677446 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:31.658239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:57345)
0923 20:22:31.659011 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:31.659054 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:31.660711 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:31.661499 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:31.661508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:31.662083 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:31.662872 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:31.662897 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:31.664064 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:31.664067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:31.665266 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:31.665278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:31.666396 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:31.666401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:31.666602 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:31.667358 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:31.667380 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:31.669733 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:31.672695 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:31.672708 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:31.672728 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:31.673369 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:31.673711 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:31.673719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:31.673726 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:31.673971 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:31.674458 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:31.674468 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:31.674940 (+ 472us) client_negotiation.cc:769] Sending connection context
0923 20:22:31.675232 (+ 292us) client_negotiation.cc:240] Negotiation successful
0923 20:22:31.675827 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":326,"threads_started":1}
I20260923 20:22:31.679037 23326 heartbeater.cc:347] Connected to a master server at 127.14.127.254:36301
I20260923 20:22:31.679358 23326 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:31.680135 23326 heartbeater.cc:511] Master 127.14.127.254:36301 requested a full tablet report, sending...
I20260923 20:22:31.682717 23137 ts_manager.cc:194] Registered new tserver with Master: 93bb1e6aa5e34f96925bd13c1ee8de75 (127.14.127.193:46059)
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:22:31.684178 23137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:57345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:22:31.711440 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36301
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:32.037328 23332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:32.037600 23332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:32.037763 23332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:32.045084 23332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:32.045197 23332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:32.045375 23332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:32.055608 23332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:32.056944 23332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:32.058238 23332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:32.072194 23338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:32.072688 23340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:32.072805 23337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:32.073494 23332 server_base.cc:1034] running on GCE node
I20260923 20:22:32.074107 23332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:32.075176 23332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:32.076418 23332 hybrid_clock.cc:648] HybridClock initialized: now 1790194952076355 us; error 59 us; skew 500 ppm
I20260923 20:22:32.076730 23332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:32 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194952, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:32.079926 23332 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:22:32.081326 23332 webserver.cc:460] Webserver started at http://127.14.127.194:45897/ using document root <none> and password file <none>
I20260923 20:22:32.082003 23332 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:32.082129 23332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:32.082456 23332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:32.085309 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "51632a66f8154182a89a220ac709a0f0"
format_stamp: "Formatted at 2026-09-23 20:22:32 on dist-test-slave-wmd0"
server_key: "788bf9d9e9c1fd3d218cb319aa7f00cf"
server_key_iv: "24b8e7b9f01594f0a7a23371182637ce"
server_key_version: "encryptionkey@0"
I20260923 20:22:32.086081 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "51632a66f8154182a89a220ac709a0f0"
format_stamp: "Formatted at 2026-09-23 20:22:32 on dist-test-slave-wmd0"
server_key: "788bf9d9e9c1fd3d218cb319aa7f00cf"
server_key_iv: "24b8e7b9f01594f0a7a23371182637ce"
server_key_version: "encryptionkey@0"
I20260923 20:22:32.091269 23332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:22:32.094775 23347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:32.096120 23332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:32.096324 23332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "51632a66f8154182a89a220ac709a0f0"
format_stamp: "Formatted at 2026-09-23 20:22:32 on dist-test-slave-wmd0"
server_key: "788bf9d9e9c1fd3d218cb319aa7f00cf"
server_key_iv: "24b8e7b9f01594f0a7a23371182637ce"
server_key_version: "encryptionkey@0"
I20260923 20:22:32.096503 23332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:32.117091 23332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:32.121387 23332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:32.121683 23332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:32.122725 23332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:32.124075 23332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:32.124150 23332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:32.124312 23332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:32.124362 23332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:32.157377 23332 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:43671
I20260923 20:22:32.157460 23459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:43671 every 8 connection(s)
I20260923 20:22:32.158651 23332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:32.160846 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 23332
I20260923 20:22:32.161015 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:32.161561 14847 external_mini_cluster.cc:1428] Setting key 52a1d3f3c3ebd7170ba6993380552ae5
Sep 23 20:22:32 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194952, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:32.181514 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:32.161347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:47581 (local address 127.14.127.254:36301)
0923 20:22:32.161566 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:32.161573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:32.162996 (+ 1423us) server_negotiation.cc:407] Connection header received
0923 20:22:32.165538 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:32.165542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:32.165603 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:32.165688 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:32.167855 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.168635 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.171218 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.171988 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.172141 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:32.175721 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:32.175726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:32.175729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:32.175760 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:32.178436 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:32.179180 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:32.179190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:32.179193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:32.179318 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:32.179746 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:32.179750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:32.179752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:32.180103 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:32.180257 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:32.180942 (+ 685us) server_negotiation.cc:299] Negotiation successful
0923 20:22:32.181168 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:22:32.182474 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:32.161957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:47581)
0923 20:22:32.162844 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:32.162887 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:32.165228 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:32.165973 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:32.165988 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:32.166834 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:32.167653 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:32.167682 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.168811 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.168826 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:32.171044 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:32.171062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.172204 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.172208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:32.172409 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:32.173094 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:32.173121 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:32.175485 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:32.178576 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:32.178584 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:32.178601 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:32.179051 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:32.179428 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:32.179434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:32.179438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:32.179623 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:32.180230 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:32.180238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:32.180581 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 20:22:32.180841 (+ 260us) client_negotiation.cc:240] Negotiation successful
0923 20:22:32.181364 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":356,"threads_started":1}
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
I20260923 20:22:32.184196 23460 heartbeater.cc:347] Connected to a master server at 127.14.127.254:36301
I20260923 20:22:32.184522 23460 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:32.185297 23460 heartbeater.cc:511] Master 127.14.127.254:36301 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
I20260923 20:22:32.186863 23137 ts_manager.cc:194] Registered new tserver with Master: 51632a66f8154182a89a220ac709a0f0 (127.14.127.194:43671)
I20260923 20:22:32.187743 23137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:47581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:22:32.213438 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36301
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:32.532830 23465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:32.533102 23465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:32.533257 23465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:32.540328 23465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:32.540436 23465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:32.540613 23465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:32.551002 23465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:32.552263 23465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:32.553586 23465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:32.566551 23470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:32.566661 23471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:32.566639 23473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:32.567811 23465 server_base.cc:1034] running on GCE node
I20260923 20:22:32.568384 23465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:32.569284 23465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:32.570510 23465 hybrid_clock.cc:648] HybridClock initialized: now 1790194952570468 us; error 42 us; skew 500 ppm
I20260923 20:22:32.570851 23465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:32 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194952, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:32.573935 23465 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:22:32.575520 23465 webserver.cc:460] Webserver started at http://127.14.127.195:44823/ using document root <none> and password file <none>
I20260923 20:22:32.576215 23465 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:32.576351 23465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:32.576674 23465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:32.579550 23465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "ad87ed08acc64aa19e39a3e516207ebf"
format_stamp: "Formatted at 2026-09-23 20:22:32 on dist-test-slave-wmd0"
server_key: "b4895d6e3713c93efe3c6004121d8a56"
server_key_iv: "bddbb19d78414138979054b659b9f19b"
server_key_version: "encryptionkey@0"
I20260923 20:22:32.580324 23465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "ad87ed08acc64aa19e39a3e516207ebf"
format_stamp: "Formatted at 2026-09-23 20:22:32 on dist-test-slave-wmd0"
server_key: "b4895d6e3713c93efe3c6004121d8a56"
server_key_iv: "bddbb19d78414138979054b659b9f19b"
server_key_version: "encryptionkey@0"
I20260923 20:22:32.585495 23465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:22:32.588943 23480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:32.590233 23465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:22:32.590413 23465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "ad87ed08acc64aa19e39a3e516207ebf"
format_stamp: "Formatted at 2026-09-23 20:22:32 on dist-test-slave-wmd0"
server_key: "b4895d6e3713c93efe3c6004121d8a56"
server_key_iv: "bddbb19d78414138979054b659b9f19b"
server_key_version: "encryptionkey@0"
I20260923 20:22:32.590616 23465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:32.623021 23465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:32.627139 23465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:32.627454 23465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:32.628415 23465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:32.629755 23465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:32.629878 23465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:32.630036 23465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:32.630108 23465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:32.664160 23465 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:45807
I20260923 20:22:32.664194 23592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:45807 every 8 connection(s)
I20260923 20:22:32.665418 23465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:32.673715 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 23465
I20260923 20:22:32.673919 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:32.674342 14847 external_mini_cluster.cc:1428] Setting key 9ea377441d39e314d4164a2e3837a07c
Sep 23 20:22:32 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194952, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:32.684684 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:32.668026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:50343 (local address 127.14.127.254:36301)
0923 20:22:32.668205 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:32.668210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:32.669264 (+ 1054us) server_negotiation.cc:407] Connection header received
0923 20:22:32.671098 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:32.671104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:32.671216 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:32.671330 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:32.672760 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.673511 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.675353 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.675911 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.676014 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:32.679509 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:32.679513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:32.679517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:32.679547 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:32.681910 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:32.682586 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:32.682590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:32.682592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:32.682771 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:32.683268 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:32.683271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:32.683273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:32.683543 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:32.683653 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:32.684276 (+ 623us) server_negotiation.cc:299] Negotiation successful
0923 20:22:32.684441 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:22:32.685581 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:32.668451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:50343)
0923 20:22:32.669099 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:32.669145 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:32.670832 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:32.671553 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:32.671561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:32.672053 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:32.672593 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:32.672610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.673679 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.673684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:32.675197 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:32.675211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.676136 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.676142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:32.676424 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:32.677110 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:32.677131 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:32.679327 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:32.682069 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:32.682076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:32.682087 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:32.682478 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:32.682868 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:32.682871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:32.682874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:32.683153 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:32.683649 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:32.683654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:32.683969 (+ 315us) client_negotiation.cc:769] Sending connection context
0923 20:22:32.684182 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 20:22:32.684531 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":261,"threads_started":1}
I20260923 20:22:32.686993 23593 heartbeater.cc:347] Connected to a master server at 127.14.127.254:36301
I20260923 20:22:32.687172 23326 heartbeater.cc:503] Master 127.14.127.254:36301 was elected leader, sending a full tablet report...
I20260923 20:22:32.687310 23593 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:32.688252 23593 heartbeater.cc:511] Master 127.14.127.254:36301 requested a full tablet report, sending...
I20260923 20:22:32.689778 23137 ts_manager.cc:194] Registered new tserver with Master: ad87ed08acc64aa19e39a3e516207ebf (127.14.127.195:45807)
I20260923 20:22:32.690543 23137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:50343
I20260923 20:22:32.701913 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab.
Entry for principal oryx/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab.
WARNING: no policy specified for oryx/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab.
Entry for principal oryx/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab.
WARNING: no policy specified for oryx/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab.
Entry for principal oryx/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab.
I20260923 20:22:32.839579 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23198
Sep 23 20:22:32 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:22:32.855945 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:32.840748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.14.127.193:46059)
0923 20:22:32.841576 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:32.841583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:32.841611 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:22:32.841674 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:32.841680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:32.841920 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:32.842057 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:32.843517 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.844559 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.845844 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.846411 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.846515 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:32.850701 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:32.850723 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:32.850738 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:32.850786 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:32.853486 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:32.854098 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:32.854105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:32.854108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:32.854231 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:32.854565 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:32.854568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:32.854570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:32.854970 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:32.855187 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:32.855434 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 20:22:32.855635 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":437,"threads_started":1}
I20260923 20:22:32.857259 23321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:32.941032 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 23198
I20260923 20:22:32.957914 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23332
Sep 23 20:22:32 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:22:32.973042 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:32.958880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.14.127.194:43671)
0923 20:22:32.959464 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:32.959471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:32.959497 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:22:32.959556 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:32.959563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:32.959768 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:32.959904 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:32.961082 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.962093 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.963330 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:32.963856 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:32.963951 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:32.967945 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:32.967956 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:32.967969 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:32.968007 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:32.970572 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:32.971237 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:32.971244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:32.971246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:32.971376 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:32.971676 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:32.971679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:32.971681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:32.971991 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:32.972198 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:32.972528 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 20:22:32.972742 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":304,"threads_started":1}
I20260923 20:22:32.974509 23455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:33.056491 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 23332
I20260923 20:22:33.072316 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23465
Sep 23 20:22:33 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:22:33.088467 23607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:33.073398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.14.127.195:45807)
0923 20:22:33.074064 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:33.074070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:33.074098 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:22:33.074159 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:33.074165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:33.074395 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:33.074544 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:33.075863 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:33.076917 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:33.078060 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:33.078597 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:33.078890 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:33.083197 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:33.083207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:33.083221 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:33.083252 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:33.085884 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:33.086469 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:33.086475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:33.086477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:33.086594 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:33.087006 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:33.087009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:33.087011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:33.087340 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:33.087571 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:33.087767 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 20:22:33.087993 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":328,"threads_started":1}
I20260923 20:22:33.089746 23588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:33.172058 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 23465
I20260923 20:22:33.187964 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23105
I20260923 20:22:33.189052 23167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:33.291148 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 23105
I20260923 20:22:33.312256 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36301
--webserver_interface=127.14.127.254
--webserver_port=35677
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:36301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:33.635022 23610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:33.635304 23610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:33.635393 23610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:33.642457 23610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:33.642554 23610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:33.642601 23610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:33.642719 23610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:33.642764 23610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:33.652381 23610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:36301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=35677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:33.653621 23610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:33.654917 23610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:33.666540 23615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:33.666672 23618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:33.666652 23616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:33.667559 23610 server_base.cc:1034] running on GCE node
I20260923 20:22:33.668443 23610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:33.669907 23610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:33.671156 23610 hybrid_clock.cc:648] HybridClock initialized: now 1790194953671052 us; error 68 us; skew 500 ppm
I20260923 20:22:33.671519 23610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:33 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194953, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:33.674940 23610 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:22:33.676553 23610 webserver.cc:460] Webserver started at http://127.14.127.254:35677/ using document root <none> and password file <none>
I20260923 20:22:33.677335 23610 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:33.677457 23610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:33.683281 23610 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 20:22:33.686196 23625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:33.687964 23610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:33.688113 23610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "389cc8e2af9f4b889d513dfc34a7b59b"
format_stamp: "Formatted at 2026-09-23 20:22:31 on dist-test-slave-wmd0"
server_key: "4e3fbc00b9452228e8f1e1d2ca685e3d"
server_key_iv: "b1137907d832c372f50278c2c2180978"
server_key_version: "encryptionkey@0"
I20260923 20:22:33.689132 23610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:33.715871 23610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:33.720222 23610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:33.720466 23610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:33.739811 23610 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:36301
I20260923 20:22:33.739830 23676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:36301 every 8 connection(s)
I20260923 20:22:33.741425 23610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:33.742985 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 23610
I20260923 20:22:33.744002 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:46059
--local_ip_for_outbound_sockets=127.14.127.193
--tserver_master_addrs=127.14.127.254:36301
--webserver_port=34201
--webserver_interface=127.14.127.193
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.127.193
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:22:33.751456 23677 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:22:33.754810 23677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Bootstrap starting.
I20260923 20:22:33.773998 23677 log.cc:824] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:33.782778 23677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:22:33.783324 23677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Bootstrap complete.
I20260923 20:22:33.792456 23677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } }
I20260923 20:22:33.793452 23677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389cc8e2af9f4b889d513dfc34a7b59b, State: Initialized, Role: FOLLOWER
I20260923 20:22:33.794159 23677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } }
I20260923 20:22:33.794415 23677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:33.794545 23677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:33.794770 23677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:22:33.798574 23677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } }
I20260923 20:22:33.799160 23677 leader_election.cc:302] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 389cc8e2af9f4b889d513dfc34a7b59b; no voters:
I20260923 20:22:33.799830 23677 leader_election.cc:288] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:22:33.799983 23681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:22:33.801205 23681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [term 2 LEADER]: Becoming Leader. State: Replica: 389cc8e2af9f4b889d513dfc34a7b59b, State: Running, Role: LEADER
I20260923 20:22:33.801884 23681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } }
I20260923 20:22:33.802103 23677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:33.804584 23683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389cc8e2af9f4b889d513dfc34a7b59b. Latest consensus state: current_term: 2 leader_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } } }
I20260923 20:22:33.804961 23683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:33.804622 23682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389cc8e2af9f4b889d513dfc34a7b59b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 36301 } } }
I20260923 20:22:33.805073 23682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:33.816139 23686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:33.822029 23686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:33.823489 23686 catalog_manager.cc:1259] Loaded cluster ID: df0407aa9dfe43549e3074d8a62eac3b
I20260923 20:22:33.823650 23686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:33.827526 23686 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:33.828990 23686 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 389cc8e2af9f4b889d513dfc34a7b59b: Loaded TSK: 0
I20260923 20:22:33.830549 23686 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:22:34.085114 23679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:34.085427 23679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:34.085578 23679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:34.092842 23679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:34.092985 23679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:34.093166 23679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:34.103088 23679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.193.keytab
--principal=oryx/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:46059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=34201
--tserver_master_addrs=127.14.127.254:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:34.104470 23679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:34.105989 23679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:34.122725 23703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:34.123322 23704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:34.124131 23706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:34.125253 23679 server_base.cc:1034] running on GCE node
I20260923 20:22:34.125887 23679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:34.126971 23679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:34.128188 23679 hybrid_clock.cc:648] HybridClock initialized: now 1790194954128136 us; error 47 us; skew 500 ppm
I20260923 20:22:34.128535 23679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194954, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:34.131764 23679 init.cc:377] Logged in from keytab as oryx/127.14.127.193@KRBTEST.COM (short username oryx)
I20260923 20:22:34.133419 23679 webserver.cc:460] Webserver started at http://127.14.127.193:34201/ using document root <none> and password file <none>
I20260923 20:22:34.134159 23679 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:34.134307 23679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:34.139895 23679 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:22:34.142675 23713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:34.144120 23679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:22:34.144287 23679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "93bb1e6aa5e34f96925bd13c1ee8de75"
format_stamp: "Formatted at 2026-09-23 20:22:31 on dist-test-slave-wmd0"
server_key: "eeaa4b8d521460e85d66b173af771572"
server_key_iv: "01bb34485bea341b3d0360ab10a005a5"
server_key_version: "encryptionkey@0"
I20260923 20:22:34.145221 23679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:34.158290 23679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:34.162742 23679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:34.163022 23679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:34.164089 23679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:34.165512 23679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:34.165608 23679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:34.165750 23679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:34.165854 23679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:34.200309 23679 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:46059
I20260923 20:22:34.200397 23825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:46059 every 8 connection(s)
I20260923 20:22:34.201788 23679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:34.208315 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 23679
I20260923 20:22:34.209090 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:43671
--local_ip_for_outbound_sockets=127.14.127.194
--tserver_master_addrs=127.14.127.254:36301
--webserver_port=45897
--webserver_interface=127.14.127.194
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.127.194
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:34.221380 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:34.204896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:42255)
0923 20:22:34.205665 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:34.205701 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:34.207492 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:34.208930 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:34.208943 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:34.209734 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:34.210520 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.210548 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.212100 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.212108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.214348 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.214366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.215443 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.215453 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.215668 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.216347 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:34.216367 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:34.220489 (+ 4122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":363,"threads_started":1}
W20260923 20:22:34.222055 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:34.204868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:42255 (local address 127.14.127.254:36301)
0923 20:22:34.205570 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:34.205586 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:34.205826 (+ 240us) server_negotiation.cc:407] Connection header received
0923 20:22:34.208016 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:34.208038 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:34.208367 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:34.208714 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:34.210745 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.211908 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.214492 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.215188 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.215333 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.220914 (+ 5581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:42255: BlockingRecv error: recv got EOF from 127.14.127.193:42255 (error 108)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":243,"threads_started":1}
W20260923 20:22:34.222148 23826 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:36301 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.127.254:36301: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:34.232137 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:34.222804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:57803)
0923 20:22:34.223019 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:34.223040 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:34.223143 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:34.223455 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:34.223463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:34.223927 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:34.224130 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.224144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.225217 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.225225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.227009 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.227027 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.227737 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.227755 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.227977 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.228594 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:34.228622 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:34.231938 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 20:22:34.232494 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:34.222958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:57803 (local address 127.14.127.254:36301)
0923 20:22:34.223095 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:34.223099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:34.223122 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:22:34.223249 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:34.223252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:34.223313 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:34.223389 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:34.224316 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.225069 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.227152 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.227582 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.227682 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.232218 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:57803: BlockingRecv error: recv got EOF from 127.14.127.193:57803 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:34.241443 23829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:34.232874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:52739)
0923 20:22:34.233092 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:34.233124 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:34.233229 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:34.233584 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:34.233591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:34.233963 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:34.234133 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.234143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.235147 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.235152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.236601 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.236615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.237248 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.237254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.237449 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.238003 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:34.238024 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:34.241163 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260923 20:22:34.241724 23826 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:22:34.241909 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:34.233154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:52739 (local address 127.14.127.254:36301)
0923 20:22:34.233322 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:34.233326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:34.233340 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:22:34.233379 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:34.233382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:34.233444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:34.233518 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:34.234261 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.235022 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.236732 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.237120 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.237210 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.241363 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:52739: BlockingRecv error: recv got EOF from 127.14.127.193:52739 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 20:22:34.535552 23830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:34.535831 23830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:34.536034 23830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:34.543320 23830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:34.543416 23830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:34.543629 23830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:34.553629 23830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.194.keytab
--principal=oryx/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:43671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=45897
--tserver_master_addrs=127.14.127.254:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:34.555015 23830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:34.556393 23830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:34.569458 23838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:34.569527 23836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:34.569707 23835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:34.570452 23830 server_base.cc:1034] running on GCE node
I20260923 20:22:34.571125 23830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:34.572026 23830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:34.573242 23830 hybrid_clock.cc:648] HybridClock initialized: now 1790194954573190 us; error 41 us; skew 500 ppm
I20260923 20:22:34.573583 23830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194954, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:34.583575 23830 init.cc:377] Logged in from keytab as oryx/127.14.127.194@KRBTEST.COM (short username oryx)
I20260923 20:22:34.585157 23830 webserver.cc:460] Webserver started at http://127.14.127.194:45897/ using document root <none> and password file <none>
I20260923 20:22:34.585930 23830 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:34.586097 23830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:34.591866 23830 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:22:34.594763 23845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:34.596180 23830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:34.596354 23830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "51632a66f8154182a89a220ac709a0f0"
format_stamp: "Formatted at 2026-09-23 20:22:32 on dist-test-slave-wmd0"
server_key: "788bf9d9e9c1fd3d218cb319aa7f00cf"
server_key_iv: "24b8e7b9f01594f0a7a23371182637ce"
server_key_version: "encryptionkey@0"
I20260923 20:22:34.597275 23830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:34.629474 23830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:34.633644 23830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:34.633931 23830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:34.634953 23830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:34.636340 23830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:34.636425 23830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:34.636582 23830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:34.636674 23830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:34.671434 23830 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:43671
I20260923 20:22:34.671561 23957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:43671 every 8 connection(s)
I20260923 20:22:34.672994 23830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:34.683545 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 23830
I20260923 20:22:34.684677 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:45807
--local_ip_for_outbound_sockets=127.14.127.195
--tserver_master_addrs=127.14.127.254:36301
--webserver_port=44823
--webserver_interface=127.14.127.195
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.127.195
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:22:34.689509 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:34.675838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:33217 (local address 127.14.127.254:36301)
0923 20:22:34.676081 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:34.676089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:34.677254 (+ 1165us) server_negotiation.cc:407] Connection header received
0923 20:22:34.679128 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:34.679133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:34.679211 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:34.679316 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:34.680777 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.681533 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.683159 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.683779 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.683921 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.689165 (+ 5244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:33217: BlockingRecv error: recv got EOF from 127.14.127.194:33217 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:22:34.689935 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:34.676395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:33217)
0923 20:22:34.677096 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:34.677132 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:34.678867 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:34.679520 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:34.679528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:34.680063 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:34.680603 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.680620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.681688 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.681692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.683015 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.683027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.683986 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.683991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.684189 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.684884 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:34.684914 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:34.688884 (+ 3970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":276,"threads_started":1}
W20260923 20:22:34.690583 23958 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:36301 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.127.254:36301: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:34.701171 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:34.691258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:59053)
0923 20:22:34.691448 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:34.691468 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:34.691553 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:34.691969 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:34.691976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:34.692425 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:34.692648 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.692659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.693972 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.693980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.695685 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.695702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.696416 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.696425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.696651 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.697265 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:34.697292 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:34.700787 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260923 20:22:34.701438 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:34.691375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:59053 (local address 127.14.127.254:36301)
0923 20:22:34.691597 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:34.691606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:34.691629 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:22:34.691693 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:34.691702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:34.691810 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:34.691914 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:34.692817 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.693825 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.695835 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.696256 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.696362 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.701029 (+ 4667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:59053: BlockingRecv error: recv got EOF from 127.14.127.194:59053 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:34 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:34.711203 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:34.702161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:38333)
0923 20:22:34.702387 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:34.702411 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:34.702539 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:34.702944 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:34.702949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:34.703396 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:34.703599 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.703609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.704654 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.704660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.705672 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:34.705683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.706457 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.706463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:34.706719 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.707470 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:34.707501 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:34.710864 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:22:34.711295 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:34.702258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:38333 (local address 127.14.127.254:36301)
0923 20:22:34.702433 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:34.702441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:34.702477 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:22:34.702721 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:34.702726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:34.702805 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:34.702917 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:34.703743 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.704505 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.705825 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:34.706293 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:34.706431 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:34.711067 (+ 4636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:38333: BlockingRecv error: recv got EOF from 127.14.127.194:38333 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 20:22:34.711604 23958 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:22:35.015285 23961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:35.015596 23961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:35.015761 23961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:35.023203 23961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:35.023353 23961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:35.023541 23961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:35.033262 23961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/krb5kdc/oryx_127.14.127.195.keytab
--principal=oryx/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:45807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=44823
--tserver_master_addrs=127.14.127.254:36301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:35.034605 23961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:35.035989 23961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:35.048725 23967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:35.048811 23966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:35.048801 23969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:35.050343 23961 server_base.cc:1034] running on GCE node
I20260923 20:22:35.051095 23961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:35.052075 23961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:35.053268 23961 hybrid_clock.cc:648] HybridClock initialized: now 1790194955053208 us; error 58 us; skew 500 ppm
I20260923 20:22:35.053609 23961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194955, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:35.056737 23961 init.cc:377] Logged in from keytab as oryx/127.14.127.195@KRBTEST.COM (short username oryx)
I20260923 20:22:35.058185 23961 webserver.cc:460] Webserver started at http://127.14.127.195:44823/ using document root <none> and password file <none>
I20260923 20:22:35.058967 23961 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:35.059123 23961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:35.064483 23961 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:22:35.067068 23976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:35.068410 23961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:22:35.068594 23961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "ad87ed08acc64aa19e39a3e516207ebf"
format_stamp: "Formatted at 2026-09-23 20:22:32 on dist-test-slave-wmd0"
server_key: "b4895d6e3713c93efe3c6004121d8a56"
server_key_iv: "bddbb19d78414138979054b659b9f19b"
server_key_version: "encryptionkey@0"
I20260923 20:22:35.069621 23961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:35.105351 23961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:35.109678 23961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:35.109969 23961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:35.110980 23961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:35.112330 23961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:35.112464 23961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:35.112607 23961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:35.112699 23961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:35.145836 23961 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:45807
I20260923 20:22:35.145924 24088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:45807 every 8 connection(s)
I20260923 20:22:35.147420 23961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:35.150424 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 23961
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:22:35.164935 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:35.149960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:44563 (local address 127.14.127.254:36301)
0923 20:22:35.150211 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:35.150220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:35.151561 (+ 1341us) server_negotiation.cc:407] Connection header received
0923 20:22:35.153688 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:35.153693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:35.153783 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:35.153896 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:35.155732 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.156820 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.158655 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.159367 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.159534 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.164654 (+ 5120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:44563: BlockingRecv error: recv got EOF from 127.14.127.195:44563 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 20:22:35.165608 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:35.150520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:44563)
0923 20:22:35.151396 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:35.151452 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:35.153426 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:35.154176 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:35.154186 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:35.154920 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:35.155560 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.155577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.156963 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.156968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.158487 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.158498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.159592 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.159597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.159796 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.160421 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:35.160446 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:35.164476 (+ 4030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":327,"threads_started":1}
W20260923 20:22:35.166288 24089 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:36301 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.127.254:36301: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
I20260923 20:22:35.167810 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:35.151681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.14.127.254:36301)
0923 20:22:35.152166 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:35.152172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:35.152190 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:22:35.152249 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:35.152255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:35.152346 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:35.152462 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:35.153990 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.155129 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.156830 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.157453 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.157583 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.160738 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:35.160751 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:35.160771 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:35.160817 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:35.163966 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:35.164787 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:35.164795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:35.164801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:35.164965 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:35.165454 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:35.165461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:35.165464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:35.166050 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:35.166355 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:35.166868 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 20:22:35.167537 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":284,"threads_started":1}
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:35.174535 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:35.166927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:36051)
0923 20:22:35.167109 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:35.167128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:35.167238 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:35.167547 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:35.167551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:35.167879 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:35.168058 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.168067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.169148 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.169151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.170231 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.170242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.171076 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.171082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.171252 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.171698 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:35.171713 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:35.174344 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 20:22:35.174957 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:35.167000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:36051 (local address 127.14.127.254:36301)
0923 20:22:35.167178 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:35.167185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:35.167205 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:22:35.167355 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:35.167360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:35.167426 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:35.167524 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:35.168180 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.169009 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.170372 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.170932 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.171056 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.174544 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:36051: BlockingRecv error: recv got EOF from 127.14.127.195:36051 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:35.182700 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:35.175194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:46539)
0923 20:22:35.175368 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:35.175384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:35.175460 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:35.175894 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:35.175898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:35.176191 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:35.176320 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.176327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.177312 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.177319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.178215 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.178225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.178935 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.178941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.179121 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.179665 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:35.179682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:35.182364 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 20:22:35.182857 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:35.175461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:46539 (local address 127.14.127.254:36301)
0923 20:22:35.175662 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:35.175666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:35.175679 (+ 13us) server_negotiation.cc:407] Connection header received
0923 20:22:35.175716 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:35.175719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:35.175766 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:35.175868 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:35.176474 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.177174 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.178360 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.178798 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.178924 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.182570 (+ 3646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:46539: BlockingRecv error: recv got EOF from 127.14.127.195:46539 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260923 20:22:35.183247 24089 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:35.251426 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:35.242402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:56499)
0923 20:22:35.243010 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:35.243030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:35.243347 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:35.243936 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:35.243941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:35.244488 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:35.244672 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.244682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.245736 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.245740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.246732 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.246744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.247513 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.247519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.247665 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.248307 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:35.248355 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:35.251188 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":341,"threads_started":1}
W20260923 20:22:35.251637 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:35.242501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:56499 (local address 127.14.127.254:36301)
0923 20:22:35.243044 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:35.243052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:35.243104 (+ 52us) server_negotiation.cc:407] Connection header received
0923 20:22:35.243555 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:35.243561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:35.243646 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:35.243746 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:35.244836 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.245575 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.246869 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.247293 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.247414 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.251428 (+ 4014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:56499: BlockingRecv error: recv got EOF from 127.14.127.193:56499 (error 108)
Metrics: {"server-negotiator.queue_time_us":425}
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:35 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:35.720333 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:35.712186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:47277)
0923 20:22:35.712680 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:35.712696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:35.712821 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:35.713102 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:35.713105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:35.713555 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:35.713739 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.713748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.714730 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.714733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.715668 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:35.715679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.716426 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.716432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:35.716604 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.717151 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:35.717168 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:35.719986 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":283,"threads_started":1}
W20260923 20:22:35.720436 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:35.712370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:47277 (local address 127.14.127.254:36301)
0923 20:22:35.712592 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:35.712597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:35.712771 (+ 174us) server_negotiation.cc:407] Connection header received
0923 20:22:35.712916 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:35.712920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:35.712981 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:35.713059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:35.713874 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.714587 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.715800 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:35.716230 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:35.716375 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:35.720187 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:47277: BlockingRecv error: recv got EOF from 127.14.127.194:47277 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 20:22:36 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:36 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:36.192417 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:36.183859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:58419)
0923 20:22:36.184376 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:36.184396 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:36.184555 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:36.184896 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:36.184901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:36.185399 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:36.185581 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:36.185590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.186595 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.186604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:36.187600 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:36.187611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.188305 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.188311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:36.188493 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:36.189247 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:36.189263 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:36.192081 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":271,"threads_started":1}
W20260923 20:22:36.192513 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:36.183946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:58419 (local address 127.14.127.254:36301)
0923 20:22:36.184198 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:36.184204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:36.184486 (+ 282us) server_negotiation.cc:407] Connection header received
0923 20:22:36.184684 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:36.184693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:36.184754 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:36.184862 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:36.185730 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.186452 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.187738 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.188120 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.188246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:36.192274 (+ 4028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:58419: BlockingRecv error: recv got EOF from 127.14.127.195:58419 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 23 20:22:36 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:36 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:36.260718 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:36.252282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:54317)
0923 20:22:36.252823 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:36.252839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:36.253012 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:36.253390 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:36.253396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:36.253802 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:36.253984 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:36.253993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.255088 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.255095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:36.256009 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:36.256021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.256751 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.256758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:36.256927 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:36.257500 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:36.257517 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:36.260480 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":303,"threads_started":1}
W20260923 20:22:36.260916 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:36.252378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:54317 (local address 127.14.127.254:36301)
0923 20:22:36.252648 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:36.252655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:36.252931 (+ 276us) server_negotiation.cc:407] Connection header received
0923 20:22:36.253140 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:36.253149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:36.253220 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:36.253321 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:36.254132 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.254962 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.256164 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.256592 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.256718 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:36.260663 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:54317: BlockingRecv error: recv got EOF from 127.14.127.193:54317 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 23 20:22:36 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:36 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:36.729185 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:36.721254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:41073)
0923 20:22:36.721848 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:36.721868 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:36.721982 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:36.722262 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:36.722266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:36.722715 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:36.722903 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:36.722912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.723884 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.723888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:36.724756 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:36.724766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.725400 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.725404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:36.725509 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:36.726124 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:36.726141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:36.728949 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":329,"threads_started":1}
W20260923 20:22:36.729367 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:36.721366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:41073 (local address 127.14.127.254:36301)
0923 20:22:36.721541 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:36.721546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:36.721939 (+ 393us) server_negotiation.cc:407] Connection header received
0923 20:22:36.722062 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:36.722065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:36.722124 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:36.722203 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:36.723020 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.723746 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.724884 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:36.725271 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:36.725370 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:36.729134 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:41073: BlockingRecv error: recv got EOF from 127.14.127.194:41073 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 20:22:37 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:37 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:37.203136 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:37.193492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:48595)
0923 20:22:37.194118 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:37.194139 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:37.194281 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:37.194726 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:37.194731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:37.195148 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:37.195332 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:37.195342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.196359 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.196363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:37.197297 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:37.197308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.198214 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.198220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:37.198676 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:37.199452 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:37.199475 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:37.202860 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":390,"threads_started":1}
W20260923 20:22:37.203435 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:37.193608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:48595 (local address 127.14.127.254:36301)
0923 20:22:37.193776 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:37.193781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:37.194219 (+ 438us) server_negotiation.cc:407] Connection header received
0923 20:22:37.194400 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:37.194403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:37.194467 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:37.194554 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:37.195489 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.196215 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.197436 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.198059 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.198199 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:37.203085 (+ 4886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:48595: BlockingRecv error: recv got EOF from 127.14.127.195:48595 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 20:22:37 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:37 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:37.269763 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:37.261799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:57707)
0923 20:22:37.262286 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:37.262302 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:37.262414 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:37.262778 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:37.262781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:37.263117 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:37.263283 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:37.263292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.264283 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.264287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:37.265152 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:37.265162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.265782 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.265786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:37.265908 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:37.266538 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:37.266555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:37.269511 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
W20260923 20:22:37.269953 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:37.261833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:57707 (local address 127.14.127.254:36301)
0923 20:22:37.262076 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:37.262084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:37.262392 (+ 308us) server_negotiation.cc:407] Connection header received
0923 20:22:37.262529 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:37.262533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:37.262595 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:37.262730 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:37.263413 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.264162 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.265272 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.265660 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.265751 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:37.269709 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:57707: BlockingRecv error: recv got EOF from 127.14.127.193:57707 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 20:22:37 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:37 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:37.738533 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:37.730158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:60159)
0923 20:22:37.730700 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:37.730737 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:37.730906 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:37.731264 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:37.731269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:37.731651 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:37.731820 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:37.731830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.732889 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.732896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:37.733802 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:37.733813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.734518 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.734523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:37.734733 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:37.735327 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:37.735344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:37.738156 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":251,"threads_started":1}
W20260923 20:22:37.738654 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:37.730268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:60159 (local address 127.14.127.254:36301)
0923 20:22:37.730496 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:37.730503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:37.730860 (+ 357us) server_negotiation.cc:407] Connection header received
0923 20:22:37.731063 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:37.731068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:37.731131 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:37.731232 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:37.731972 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.732746 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.733947 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:37.734363 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:37.734493 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:37.738377 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:60159: BlockingRecv error: recv got EOF from 127.14.127.194:60159 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 23 20:22:38 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:38 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:38.213402 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:38.204071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:59457)
0923 20:22:38.204791 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:38.204815 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:38.205006 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:38.205435 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:38.205440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:38.205848 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:38.206022 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:38.206032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.207236 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.207241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:38.208233 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:38.208245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.208998 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.209004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:38.209199 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:38.209949 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:38.209969 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:38.213070 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":330,"threads_started":1}
W20260923 20:22:38.213579 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:38.204248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:59457 (local address 127.14.127.254:36301)
0923 20:22:38.204524 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:38.204530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:38.204922 (+ 392us) server_negotiation.cc:407] Connection header received
0923 20:22:38.205163 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:38.205171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:38.205275 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:38.205396 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:38.206183 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.207027 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.208398 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.208836 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.208973 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:38.213354 (+ 4381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:59457: BlockingRecv error: recv got EOF from 127.14.127.195:59457 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 23 20:22:38 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:38 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:38.279526 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:38.270801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:32851)
0923 20:22:38.271307 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:38.271323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:38.271443 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:38.271730 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:38.271733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:38.272087 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:38.272231 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:38.272239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.273545 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.273551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:38.274912 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:38.274924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.275817 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.275822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:38.275954 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:38.276616 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:38.276634 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:38.279288 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":280,"threads_started":1}
W20260923 20:22:38.279747 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:38.270939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:32851 (local address 127.14.127.254:36301)
0923 20:22:38.271146 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:38.271150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:38.271404 (+ 254us) server_negotiation.cc:407] Connection header received
0923 20:22:38.271536 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:38.271539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:38.271599 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:38.271681 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:38.272400 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.273370 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.275086 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.275680 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.275780 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:38.279511 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:32851: BlockingRecv error: recv got EOF from 127.14.127.193:32851 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 20:22:38 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:38 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:38.748363 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:38.739501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:33777)
0923 20:22:38.739993 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:38.740011 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:38.740155 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:38.740559 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:38.740565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:38.740956 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:38.741113 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:38.741121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.742190 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.742195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:38.743299 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:38.743312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.744079 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.744085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:38.744278 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:38.744946 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:38.744964 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:38.748044 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":351,"threads_started":1}
W20260923 20:22:38.748661 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:38.739649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:33777 (local address 127.14.127.254:36301)
0923 20:22:38.739855 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:38.739862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:38.740084 (+ 222us) server_negotiation.cc:407] Connection header received
0923 20:22:38.740298 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:38.740303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:38.740388 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:38.740503 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:38.741258 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.742043 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.743449 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:38.743914 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:38.744060 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:38.748375 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:33777: BlockingRecv error: recv got EOF from 127.14.127.194:33777 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 20:22:39 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:39 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:39.227348 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:39.214346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:57427)
0923 20:22:39.214983 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:39.215002 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:39.215115 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:39.215410 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:39.215414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:39.215756 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:39.215911 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:39.215918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.217179 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.217186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:39.218579 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:39.218591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.219298 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.219304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:39.219416 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:39.224187 (+ 4771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:39.224205 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:39.227103 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":396,"threads_started":1}
W20260923 20:22:39.227532 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:39.214483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:57427 (local address 127.14.127.254:36301)
0923 20:22:39.214724 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:39.214729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:39.215077 (+ 348us) server_negotiation.cc:407] Connection header received
0923 20:22:39.215208 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:39.215212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:39.215271 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:39.215354 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:39.216040 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.216967 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.218763 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.219178 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.219268 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:39.227305 (+ 8037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:57427: BlockingRecv error: recv got EOF from 127.14.127.195:57427 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 20:22:39 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:39 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:39.288969 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:39.280440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:38361)
0923 20:22:39.280962 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:39.280977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:39.281130 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:39.281494 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:39.281499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:39.281941 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:39.282092 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:39.282101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.283231 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.283241 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:39.284192 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:39.284204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.284891 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.284897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:39.285076 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:39.285629 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:39.285646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:39.288646 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":251,"threads_started":1}
W20260923 20:22:39.289304 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:39.280516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:38361 (local address 127.14.127.254:36301)
0923 20:22:39.280778 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:39.280785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:39.281060 (+ 275us) server_negotiation.cc:407] Connection header received
0923 20:22:39.281276 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:39.281283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:39.281361 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:39.281466 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:39.282251 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.283078 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.284343 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.284748 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.284878 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:39.288903 (+ 4025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:38361: BlockingRecv error: recv got EOF from 127.14.127.193:38361 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 20:22:39 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:39 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:39.761785 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:39.749483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:34649)
0923 20:22:39.749970 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:39.749987 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:39.750094 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:39.750455 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:39.750458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:39.750986 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:39.751228 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:39.751242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.752565 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.752569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:39.753466 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:39.753477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.754198 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.754203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:39.754330 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:39.758756 (+ 4426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:39.758774 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:39.761539 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1}
W20260923 20:22:39.762166 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:39.749659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:34649 (local address 127.14.127.254:36301)
0923 20:22:39.749889 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:39.749898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:39.750061 (+ 163us) server_negotiation.cc:407] Connection header received
0923 20:22:39.750233 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:39.750239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:39.750303 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:39.750410 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:39.751464 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.752433 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.753617 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:39.754034 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:39.754141 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:39.761740 (+ 7599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:34649: BlockingRecv error: recv got EOF from 127.14.127.194:34649 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 23 20:22:40 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:40 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:40.236864 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:40.228366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:45387)
0923 20:22:40.228925 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:40.228945 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:40.229123 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:40.229480 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:40.229485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:40.229924 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:40.230080 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:40.230088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.231180 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.231187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:40.232227 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:40.232239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.232972 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.232978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:40.233148 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:40.233702 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:40.233719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:40.236506 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":291,"threads_started":1}
W20260923 20:22:40.236979 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:40.228489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:45387 (local address 127.14.127.254:36301)
0923 20:22:40.228741 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:40.228748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:40.229054 (+ 306us) server_negotiation.cc:407] Connection header received
0923 20:22:40.229254 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:40.229259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:40.229335 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:40.229445 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:40.230237 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.231039 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.232388 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.232826 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.232957 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:40.236715 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:45387: BlockingRecv error: recv got EOF from 127.14.127.195:45387 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 23 20:22:40 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:40 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:22:40.299044 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:40.290120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:39085 (local address 127.14.127.254:36301)
0923 20:22:40.290388 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:40.290395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:40.290760 (+ 365us) server_negotiation.cc:407] Connection header received
0923 20:22:40.290965 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:40.290971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:40.291058 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:40.291149 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:40.292235 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.293125 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.294513 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.294942 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.295034 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:40.298838 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:39085: BlockingRecv error: recv got EOF from 127.14.127.193:39085 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:22:40.299069 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:40.290068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:39085)
0923 20:22:40.290644 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:40.290665 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:40.290820 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:40.291228 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:40.291233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:40.291835 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:40.292038 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:40.292053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.293380 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.293386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:40.294365 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:40.294377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.295096 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.295100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:40.295256 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:40.295905 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:40.295922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:40.298676 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":278,"threads_started":1}
Sep 23 20:22:40 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:40 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:40.770776 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:40.762748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:52593)
0923 20:22:40.763217 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:40.763234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:40.763367 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:40.763651 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:40.763654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:40.764049 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:40.764198 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:40.764207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.765201 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.765206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:40.766170 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:40.766183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.766928 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.766933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:40.767065 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:40.767650 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:40.767667 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:40.770490 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":258,"threads_started":1}
W20260923 20:22:40.771026 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:40.762830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:52593 (local address 127.14.127.254:36301)
0923 20:22:40.763056 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:40.763062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:40.763321 (+ 259us) server_negotiation.cc:407] Connection header received
0923 20:22:40.763464 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:40.763467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:40.763528 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:40.763607 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:40.764368 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.765064 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.766328 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:40.766753 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:40.766845 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:40.770786 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:52593: BlockingRecv error: recv got EOF from 127.14.127.194:52593 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 20:22:41 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:41 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:41.246109 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:41.237921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:55867)
0923 20:22:41.238388 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:41.238420 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:41.238581 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:41.238945 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:41.238949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:41.239480 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:41.239686 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:41.239698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.240965 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.240969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:41.241849 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:41.241859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.242544 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.242548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:41.242686 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:41.243225 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:41.243241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:41.245896 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":267,"threads_started":1}
W20260923 20:22:41.246361 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:41.238084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:55867 (local address 127.14.127.254:36301)
0923 20:22:41.238322 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:41.238328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:41.238499 (+ 171us) server_negotiation.cc:407] Connection header received
0923 20:22:41.238719 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:41.238723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:41.238785 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:41.238907 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:41.239858 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.240841 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.241979 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.242401 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.242508 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:41.246090 (+ 3582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:55867: BlockingRecv error: recv got EOF from 127.14.127.195:55867 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 23 20:22:41 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:41 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:41.308435 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:41.300131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:40955)
0923 20:22:41.300800 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:41.300816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:41.300938 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:41.301274 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:41.301277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:41.301655 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:41.301826 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:41.301834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.302928 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.302933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:41.303899 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:41.303910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.304549 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.304553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:41.304660 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:41.305202 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:41.305219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:41.308189 (+ 2970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":271,"threads_started":1}
W20260923 20:22:41.308672 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:41.300221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:40955 (local address 127.14.127.254:36301)
0923 20:22:41.300515 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:41.300524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:41.300899 (+ 375us) server_negotiation.cc:407] Connection header received
0923 20:22:41.301075 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:41.301079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:41.301142 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:41.301225 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:41.301981 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.302777 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.304021 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.304424 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.304516 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:41.308404 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:40955: BlockingRecv error: recv got EOF from 127.14.127.193:40955 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 23 20:22:41 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:41 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:41.781208 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:41.771827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:42581)
0923 20:22:41.772514 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:41.772531 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:41.772769 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:41.773146 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:41.773151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:41.773613 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:41.773844 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:41.773854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.774993 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.775000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:41.775891 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:41.775902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.776640 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.776647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:41.776813 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:41.777572 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:41.777595 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:41.780858 (+ 3263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":347,"threads_started":1}
W20260923 20:22:41.781327 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:41.771887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:42581 (local address 127.14.127.254:36301)
0923 20:22:41.772112 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:41.772117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:41.772651 (+ 534us) server_negotiation.cc:407] Connection header received
0923 20:22:41.772902 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:41.772908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:41.772997 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:41.773110 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:41.773994 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.774861 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.776033 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:41.776443 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:41.776603 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:41.781055 (+ 4452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:42581: BlockingRecv error: recv got EOF from 127.14.127.194:42581 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 23 20:22:42 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:42 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:42.255947 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:42.247151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:34265)
0923 20:22:42.247741 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:42.247760 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:42.247873 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:42.248159 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:42.248162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:42.248598 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:42.248776 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:42.248784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.250056 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.250062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:42.251228 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:42.251244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.251977 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.251983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:42.252148 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:42.252683 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:42.252700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:42.255609 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":279,"threads_started":1}
W20260923 20:22:42.256033 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:42.247219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:34265 (local address 127.14.127.254:36301)
0923 20:22:42.247423 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:42.247428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:42.247831 (+ 403us) server_negotiation.cc:407] Connection header received
0923 20:22:42.247970 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:42.247974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:42.248033 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:42.248148 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:42.248894 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.249914 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.251372 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.251815 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.251951 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:42.255800 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:34265: BlockingRecv error: recv got EOF from 127.14.127.195:34265 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 20:22:42 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:42 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:42.317816 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:42.309371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:57363)
0923 20:22:42.310003 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:42.310020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:42.310179 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:42.310526 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:42.310533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:42.310979 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:42.311124 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:42.311133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.312154 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.312160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:42.313071 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:42.313081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.313842 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.313849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:42.314011 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:42.314577 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:42.314593 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:42.317439 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":389,"threads_started":1}
W20260923 20:22:42.317940 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:42.309527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:57363 (local address 127.14.127.254:36301)
0923 20:22:42.309760 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:42.309764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:42.310105 (+ 341us) server_negotiation.cc:407] Connection header received
0923 20:22:42.310299 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:42.310305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:42.310382 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:42.310483 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:42.311265 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.312008 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.313210 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.313665 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.313805 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:42.317629 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:57363: BlockingRecv error: recv got EOF from 127.14.127.193:57363 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 20:22:42 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:42 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:42.790297 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:42.782201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:39069)
0923 20:22:42.782774 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:42.782790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:42.782922 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:42.783212 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:42.783215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:42.783593 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:42.783745 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:42.783754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.784898 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.784902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:42.785854 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:42.785867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.786555 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.786559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:42.786733 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:42.787343 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:42.787360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:42.790070 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":279,"threads_started":1}
W20260923 20:22:42.790527 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:42.782274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:39069 (local address 127.14.127.254:36301)
0923 20:22:42.782488 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:42.782493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:42.782882 (+ 389us) server_negotiation.cc:407] Connection header received
0923 20:22:42.783020 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:42.783024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:42.783086 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:42.783168 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:42.783871 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.784737 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.785991 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:42.786398 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:42.786490 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:42.790257 (+ 3767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:39069: BlockingRecv error: recv got EOF from 127.14.127.194:39069 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 20:22:43 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:43 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:43.266016 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:43.256982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:38823)
0923 20:22:43.257806 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:43.257827 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:43.257944 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:43.258255 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:43.258258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:43.258741 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:43.258897 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:43.258906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.260177 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.260181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:43.261142 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:43.261154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.261881 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.261886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:43.262004 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:43.262581 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:43.262599 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:43.265793 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":550,"threads_started":1}
W20260923 20:22:43.266278 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:43.257069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:38823 (local address 127.14.127.254:36301)
0923 20:22:43.257348 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:43.257358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:43.257927 (+ 569us) server_negotiation.cc:407] Connection header received
0923 20:22:43.258048 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:43.258052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:43.258115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:43.258203 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:43.259066 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.260033 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.261296 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.261698 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.261798 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:43.265995 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:38823: BlockingRecv error: recv got EOF from 127.14.127.195:38823 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 20:22:43 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:43 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:43.327239 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:43.318916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:44933)
0923 20:22:43.319401 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:43.319417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:43.319528 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:43.319828 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:43.319831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:43.320208 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:43.320369 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:43.320377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.321313 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.321317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:43.322242 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:43.322253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.323216 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.323222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:43.323419 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:43.324060 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:43.324082 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:43.326980 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
W20260923 20:22:43.327404 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:43.318924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:44933 (local address 127.14.127.254:36301)
0923 20:22:43.319159 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:43.319168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:43.319499 (+ 331us) server_negotiation.cc:407] Connection header received
0923 20:22:43.319626 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:43.319630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:43.319689 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:43.319767 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:43.320493 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.321208 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.322378 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.323048 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.323177 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:43.327161 (+ 3984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:44933: BlockingRecv error: recv got EOF from 127.14.127.193:44933 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 20:22:43 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:43 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:43.800237 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:43.791267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:35635)
0923 20:22:43.791857 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:43.791874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:43.791995 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:43.792420 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:43.792424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:43.792891 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:43.793060 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:43.793069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.794253 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.794257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:43.795399 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:43.795411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.796362 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.796370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:43.796540 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:43.797172 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:43.797189 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:43.799991 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":319,"threads_started":1}
W20260923 20:22:43.800537 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:43.791375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:35635 (local address 127.14.127.254:36301)
0923 20:22:43.791538 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:43.791542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:43.791951 (+ 409us) server_negotiation.cc:407] Connection header received
0923 20:22:43.792148 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:43.792152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:43.792220 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:43.792313 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:43.793219 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.794111 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.795534 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:43.796126 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:43.796267 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:43.800190 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:35635: BlockingRecv error: recv got EOF from 127.14.127.194:35635 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 20:22:44 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:44 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:44.275593 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:44.267274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:48901)
0923 20:22:44.267760 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:44.267780 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:44.267914 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:44.268191 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:44.268195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:44.268590 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:44.268749 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:44.268757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.269667 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.269670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:44.270589 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:44.270599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.271356 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.271361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:44.271492 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:44.272048 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:44.272064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:44.275356 (+ 3292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
W20260923 20:22:44.275774 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:44.267303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:48901 (local address 127.14.127.254:36301)
0923 20:22:44.267528 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:44.267537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:44.267858 (+ 321us) server_negotiation.cc:407] Connection header received
0923 20:22:44.268010 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:44.268013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:44.268073 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:44.268152 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:44.268866 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.269557 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.270750 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.271174 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.271289 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:44.275560 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:48901: BlockingRecv error: recv got EOF from 127.14.127.195:48901 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 20:22:44 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:44 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:44.336683 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:44.328214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:37247)
0923 20:22:44.328783 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:44.328809 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:44.328918 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:44.329199 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:44.329202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:44.329597 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:44.329742 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:44.329751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.330793 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.330797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:44.331739 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:44.331750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.332459 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.332465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:44.332636 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:44.333415 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:44.333434 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:44.336464 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":331,"threads_started":1}
W20260923 20:22:44.336922 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:44.328383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:37247 (local address 127.14.127.254:36301)
0923 20:22:44.328553 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:44.328557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:44.328883 (+ 326us) server_negotiation.cc:407] Connection header received
0923 20:22:44.329012 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:44.329015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:44.329072 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:44.329150 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:44.329867 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.330631 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.331859 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.332292 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.332380 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:44.336637 (+ 4257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:37247: BlockingRecv error: recv got EOF from 127.14.127.193:37247 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 20:22:44 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:44 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:44.816826 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:44.801237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:44559)
0923 20:22:44.801808 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:44.801828 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:44.801989 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:44.802547 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:44.802560 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:44.803061 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:44.803254 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:44.803264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.804420 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.804427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:44.805352 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:44.805364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.806106 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.806121 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:44.806307 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:44.812878 (+ 6571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:44.812902 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:44.816573 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":335,"threads_started":1}
W20260923 20:22:44.817034 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:44.801368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:44559 (local address 127.14.127.254:36301)
0923 20:22:44.801591 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:44.801599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:44.801924 (+ 325us) server_negotiation.cc:407] Connection header received
0923 20:22:44.802147 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:44.802159 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:44.802224 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:44.802341 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:44.803404 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.804279 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.805499 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:44.805944 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:44.806088 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:44.816807 (+ 10719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:44559: BlockingRecv error: recv got EOF from 127.14.127.194:44559 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 20:22:45 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:45 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:45.285332 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:45.276630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:44251)
0923 20:22:45.277150 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:45.277170 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:45.277282 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:45.277592 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:45.277596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:45.277960 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:45.278104 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:45.278111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.279209 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.279213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:45.280195 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:45.280208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.280890 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.280895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:45.281018 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:45.281608 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:45.281626 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:45.284961 (+ 3335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":309,"threads_started":1}
W20260923 20:22:45.285427 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:45.276867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:44251 (local address 127.14.127.254:36301)
0923 20:22:45.277095 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:45.277104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:45.277248 (+ 144us) server_negotiation.cc:407] Connection header received
0923 20:22:45.277386 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:45.277390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:45.277451 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:45.277532 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:45.278239 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.279057 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.280337 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.280753 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.280860 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:45.285181 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:44251: BlockingRecv error: recv got EOF from 127.14.127.195:44251 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 23 20:22:45 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:45 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:45.346459 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:45.337620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:41671)
0923 20:22:45.338097 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:45.338113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:45.338264 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:45.338599 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:45.338602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:45.339059 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:45.339266 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:45.339275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.340278 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.340282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:45.341166 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:45.341177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.341976 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.341981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:45.342125 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:45.342843 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:45.342861 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:45.346088 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":278,"threads_started":1}
W20260923 20:22:45.346537 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:45.337832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:41671 (local address 127.14.127.254:36301)
0923 20:22:45.338087 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:45.338092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:45.338170 (+ 78us) server_negotiation.cc:407] Connection header received
0923 20:22:45.338394 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:45.338398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:45.338471 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:45.338553 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:45.339398 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.340134 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.341333 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.341795 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.341912 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:45.346297 (+ 4385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:41671: BlockingRecv error: recv got EOF from 127.14.127.193:41671 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 23 20:22:45 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:45 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:45.825882 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:45.817697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:37247)
0923 20:22:45.818177 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:45.818193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:45.818376 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:45.818749 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:45.818755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:45.819218 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:45.819397 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:45.819407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.820423 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.820430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:45.821324 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:45.821335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.822074 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.822081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:45.822257 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:45.822873 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:45.822891 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:45.825626 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":270,"threads_started":1}
W20260923 20:22:45.826092 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:45.817846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:37247 (local address 127.14.127.254:36301)
0923 20:22:45.818046 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:45.818053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:45.818338 (+ 285us) server_negotiation.cc:407] Connection header received
0923 20:22:45.818515 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:45.818521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:45.818594 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:45.818713 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:45.819548 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.820266 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.821466 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:45.821891 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:45.822029 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:45.825841 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:37247: BlockingRecv error: recv got EOF from 127.14.127.194:37247 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 20:22:46 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:46 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:46.295527 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:46.286414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:43799)
0923 20:22:46.287023 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:46.287044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:46.287162 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:46.287485 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:46.287488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:46.287882 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:46.288056 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:46.288064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.289036 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.289040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:46.290380 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:46.290397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.291421 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.291426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:46.291556 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:46.292224 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:46.292250 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:46.295277 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":388,"threads_started":1}
W20260923 20:22:46.295717 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:46.286546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:43799 (local address 127.14.127.254:36301)
0923 20:22:46.286752 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:46.286756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:46.287126 (+ 370us) server_negotiation.cc:407] Connection header received
0923 20:22:46.287278 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:46.287283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:46.287353 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:46.287444 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:46.288195 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.288927 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.290549 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.291241 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.291367 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:46.295493 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:43799: BlockingRecv error: recv got EOF from 127.14.127.195:43799 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 20:22:46 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:46 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:46.356560 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:46.347489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:34347)
0923 20:22:46.347990 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:46.348008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:46.348134 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:46.348503 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:46.348506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:46.349048 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:46.349227 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:46.349236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.350313 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.350318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:46.351802 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:46.351817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.352733 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.352737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:46.352864 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:46.353516 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:46.353533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:46.356330 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":307,"threads_started":1}
W20260923 20:22:46.356792 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:46.347554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:34347 (local address 127.14.127.254:36301)
0923 20:22:46.347781 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:46.347791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:46.348091 (+ 300us) server_negotiation.cc:407] Connection header received
0923 20:22:46.348272 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:46.348275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:46.348359 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:46.348442 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:46.349355 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.350106 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.351959 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.352589 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.352687 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:46.356537 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:34347: BlockingRecv error: recv got EOF from 127.14.127.193:34347 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 20:22:46 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:46 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:46.835459 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:46.826781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:57845)
0923 20:22:46.827292 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:46.827315 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:46.827458 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:46.827778 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:46.827782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:46.828150 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:46.828300 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:46.828308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.829280 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.829286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:46.830297 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:46.830310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.830976 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.830981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:46.831126 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:46.831728 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:46.831748 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:46.835106 (+ 3358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
W20260923 20:22:46.835525 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:46.827237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:57845 (local address 127.14.127.254:36301)
0923 20:22:46.827459 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:46.827463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:46.827480 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:22:46.827555 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:46.827558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:46.827621 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:46.827725 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:46.828432 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.829109 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.830426 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:46.830850 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:46.830937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:46.835343 (+ 4406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:57845: BlockingRecv error: recv got EOF from 127.14.127.194:57845 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:22:47 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:47 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:47.305490 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:47.296582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:52591)
0923 20:22:47.297102 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:47.297122 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:47.297279 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:47.297648 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:47.297653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:47.298067 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:47.298217 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:47.298225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.299331 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.299337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:47.300257 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:47.300268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.301056 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.301061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:47.301238 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:47.301945 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:47.301962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:47.305243 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":309,"threads_started":1}
W20260923 20:22:47.305682 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:47.296647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:52591 (local address 127.14.127.254:36301)
0923 20:22:47.296938 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:47.296946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:47.297207 (+ 261us) server_negotiation.cc:407] Connection header received
0923 20:22:47.297433 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:47.297439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:47.297509 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:47.297613 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:47.298365 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.299177 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.300405 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.300892 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.301030 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:47.305445 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:52591: BlockingRecv error: recv got EOF from 127.14.127.195:52591 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 23 20:22:47 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:47 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:47.366396 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:47.357602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:45243)
0923 20:22:47.358149 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:47.358166 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:47.358294 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:47.358658 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:47.358665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:47.359087 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:47.359270 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:47.359280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.360284 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.360289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:47.361382 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:47.361395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.362174 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.362179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:47.362328 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:47.363049 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:47.363077 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:47.366135 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":296,"threads_started":1}
W20260923 20:22:47.366585 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:47.357714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:45243 (local address 127.14.127.254:36301)
0923 20:22:47.357981 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:47.357987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:47.358254 (+ 267us) server_negotiation.cc:407] Connection header received
0923 20:22:47.358394 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:47.358397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:47.358474 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:47.358555 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:47.359407 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.360139 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.361543 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.361999 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.362102 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:47.366356 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:45243: BlockingRecv error: recv got EOF from 127.14.127.193:45243 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 23 20:22:47 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:47 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:47.845132 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:47.836660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:39399)
0923 20:22:47.837190 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:47.837216 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:47.837380 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:47.837731 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:47.837737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:47.838175 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:47.838335 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:47.838344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.839452 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.839457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:47.840363 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:47.840375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.841091 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.841097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:47.841276 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:47.841912 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:47.841931 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:47.844892 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":331,"threads_started":1}
W20260923 20:22:47.845412 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:47.836802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:39399 (local address 127.14.127.254:36301)
0923 20:22:47.836995 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:47.837003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:47.837309 (+ 306us) server_negotiation.cc:407] Connection header received
0923 20:22:47.837505 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:47.837511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:47.837588 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:47.837693 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:47.838488 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.839302 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.840516 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:47.840934 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:47.841063 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:47.845117 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:39399: BlockingRecv error: recv got EOF from 127.14.127.194:39399 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 20:22:48 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:48 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:48.315300 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:48.306575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:56281)
0923 20:22:48.307174 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:48.307209 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:48.307397 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:48.307779 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:48.307784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:48.308225 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:48.308386 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:48.308394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.309466 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.309473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:48.310371 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:48.310382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.311211 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.311218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:48.311378 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:48.312035 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:48.312053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:48.315059 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":321,"threads_started":1}
W20260923 20:22:48.315490 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:48.306729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:56281 (local address 127.14.127.254:36301)
0923 20:22:48.306976 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:48.306983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:48.307323 (+ 340us) server_negotiation.cc:407] Connection header received
0923 20:22:48.307533 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:48.307538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:48.307613 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:48.307716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:48.308562 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.309320 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.310528 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.311017 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.311150 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:48.315288 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:56281: BlockingRecv error: recv got EOF from 127.14.127.195:56281 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 23 20:22:48 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:48 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:48.376034 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:48.367445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:58283)
0923 20:22:48.367985 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:48.368002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:48.368159 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:48.368580 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:48.368585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:48.368986 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:48.369152 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:48.369161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.370204 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.370209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:48.371221 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:48.371233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.371977 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.371983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:48.372163 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:48.372763 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:48.372781 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:48.375768 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":272,"threads_started":1}
W20260923 20:22:48.376206 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:48.367461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:58283 (local address 127.14.127.254:36301)
0923 20:22:48.367762 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:48.367768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:48.368090 (+ 322us) server_negotiation.cc:407] Connection header received
0923 20:22:48.368304 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:48.368315 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:48.368435 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:48.368543 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:48.369311 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.370051 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.371365 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.371818 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.371950 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:48.375970 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:58283: BlockingRecv error: recv got EOF from 127.14.127.193:58283 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 23 20:22:48 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:48 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:48.854348 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:48.846197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:58369)
0923 20:22:48.846700 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:48.846715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:48.846880 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:48.847227 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:48.847233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:48.847621 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:48.847801 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:48.847811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.848890 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.848894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:48.849792 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:48.849803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.850526 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.850532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:48.850741 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:48.851316 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:48.851334 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:48.854027 (+ 2693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
W20260923 20:22:48.854483 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:48.846135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:58369 (local address 127.14.127.254:36301)
0923 20:22:48.846324 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:48.846328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:48.846802 (+ 474us) server_negotiation.cc:407] Connection header received
0923 20:22:48.847004 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:48.847009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:48.847080 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:48.847193 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:48.847953 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.848672 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.849939 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:48.850374 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:48.850501 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:48.854241 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:58369: BlockingRecv error: recv got EOF from 127.14.127.194:58369 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 20:22:49 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:49 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.195@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:49.324815 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:49.316395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.195:36533)
0923 20:22:49.316905 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:49.316924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:49.317082 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:49.317474 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:49.317479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:49.317987 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:49.318157 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:49.318165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.319297 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.319304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:49.320200 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:49.320212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.320907 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.320912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:49.321072 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:49.321655 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:49.321671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:49.324452 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":302,"threads_started":1}
W20260923 20:22:49.324913 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:49.316491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:36533 (local address 127.14.127.254:36301)
0923 20:22:49.316713 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:49.316721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:49.317008 (+ 287us) server_negotiation.cc:407] Connection header received
0923 20:22:49.317252 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:49.317255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:49.317315 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:49.317419 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:49.318299 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.319155 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.320338 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.320758 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.320881 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:49.324677 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.195:36533: BlockingRecv error: recv got EOF from 127.14.127.195:36533 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:22:49 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:49 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.193@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:49.385182 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:49.377284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.193:56435)
0923 20:22:49.377744 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:49.377760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:49.377879 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:49.378185 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:49.378188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:49.378583 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:49.378815 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:49.378824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.379797 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.379800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:49.380698 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:49.380708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.381378 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.381382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:49.381495 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:49.382117 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:49.382134 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:49.384939 (+ 2805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":255,"threads_started":1}
W20260923 20:22:49.385445 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:49.377337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:56435 (local address 127.14.127.254:36301)
0923 20:22:49.377568 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:49.377576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:49.377843 (+ 267us) server_negotiation.cc:407] Connection header received
0923 20:22:49.377984 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:49.377988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:49.378070 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:49.378151 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:49.378937 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.379679 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.380832 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.381241 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.381330 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:49.385145 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.193:56435: BlockingRecv error: recv got EOF from 127.14.127.193:56435 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 23 20:22:49 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:22:49 dist-test-slave-wmd0 krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.127.194@KRBTEST.COM for oryx/127.14.127.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:22:49.875934 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:49.855611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:36301 (local address 127.14.127.194:45435)
0923 20:22:49.856112 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:49.856129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:49.856244 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:49.856553 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:49.856556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:49.856934 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:49.857097 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:49.857105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.858093 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.858097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:49.859337 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:49.859350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.860239 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.860245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:49.860408 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:49.872418 (+ 12010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:49.872439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:49.875568 (+ 3129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.127.254:36301: Server oryx/127.14.127.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":272,"threads_started":1}
W20260923 20:22:49.876066 23828 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:49.855685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:45435 (local address 127.14.127.254:36301)
0923 20:22:49.855923 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:49.855930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:49.856239 (+ 309us) server_negotiation.cc:407] Connection header received
0923 20:22:49.856344 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:49.856347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:49.856407 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:49.856489 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:49.857240 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.857960 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.859474 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:49.860096 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:49.860223 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:49.875785 (+ 15562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.127.194:45435: BlockingRecv error: recv got EOF from 127.14.127.194:45435 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:22:50.159042 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23679
I20260923 20:22:50.171394 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:50.160114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.14.127.193:46059)
0923 20:22:50.160735 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:50.160742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:50.160769 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:22:50.160831 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:50.160837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:50.161071 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:50.161220 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:50.162561 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:50.163649 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:50.164817 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:50.165358 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:50.165454 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:50.167949 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:50.167963 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:50.167979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:50.168017 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:50.170844 (+ 2827us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.127.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:22:50.171128 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57248: Request ticket server kudu/127.14.127.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":337,"threads_started":1}
W20260923 20:22:50.171509 24139 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57248: Request ticket server kudu/127.14.127.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:22:50.171602 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23679: Not authorized: Client connection negotiation failed: client connection to 127.14.127.193:46059: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.127.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:22:50.171844 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 23679
I20260923 20:22:50.188732 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23830
I20260923 20:22:50.201411 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:50.189915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.14.127.194:43671)
0923 20:22:50.190484 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:50.190492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:50.190520 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:22:50.190579 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:50.190586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:50.190903 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:50.191056 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:50.192430 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:50.193611 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:50.195011 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:50.195553 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:50.195655 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:50.198293 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:50.198302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:50.198316 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:50.198354 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:50.200855 (+ 2501us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.127.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:22:50.201105 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40040: Request ticket server kudu/127.14.127.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":273,"threads_started":1}
W20260923 20:22:50.201521 24140 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40040: Request ticket server kudu/127.14.127.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:22:50.201676 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23830: Not authorized: Client connection negotiation failed: client connection to 127.14.127.194:43671: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.127.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:22:50.201836 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 23830
I20260923 20:22:50.219321 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23961
I20260923 20:22:50.231467 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:50.220297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.14.127.195:45807)
0923 20:22:50.220926 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:50.220933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:50.220959 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:22:50.221019 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:50.221025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:50.221235 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:50.221394 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:50.222776 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:50.223835 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:50.225027 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:50.225595 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:50.225699 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:50.228325 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:50.228333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:50.228347 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:50.228379 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:50.230975 (+ 2596us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.127.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:22:50.231229 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43232: Request ticket server kudu/127.14.127.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":315,"threads_started":1}
W20260923 20:22:50.231587 24141 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43232: Request ticket server kudu/127.14.127.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:22:50.231597 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23961: Not authorized: Client connection negotiation failed: client connection to 127.14.127.195:45807: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.14.127.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:22:50.231766 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 23961
I20260923 20:22:50.247982 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 23610
I20260923 20:22:50.249265 23672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:50.366734 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 23610
2026-09-23T20:22:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21035 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T20:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:50Z Disabled control of system clock
I20260923 20:22:50.442014 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:39889
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:39889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 20:22:50.767972 24146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:50.768276 24146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:50.768355 24146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:50.775787 24146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:50.775907 24146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:50.775945 24146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:50.775995 24146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:50.776041 24146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:50.785509 24146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:39889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:39889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:50.787113 24146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:50.789108 24146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:50.801337 24151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:50.801415 24152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:50.801379 24154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:50.802331 24146 server_base.cc:1034] running on GCE node
I20260923 20:22:50.803193 24146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:50.804641 24146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:50.805869 24146 hybrid_clock.cc:648] HybridClock initialized: now 1790194970805825 us; error 50 us; skew 500 ppm
I20260923 20:22:50.806226 24146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:50.808550 24146 webserver.cc:460] Webserver started at http://127.14.127.254:37891/ using document root <none> and password file <none>
I20260923 20:22:50.809271 24146 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:50.809409 24146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:50.809741 24146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:50.812836 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "f3bade4c4eb64cf787d5f36e0e9e6999"
format_stamp: "Formatted at 2026-09-23 20:22:50 on dist-test-slave-wmd0"
server_key: "49b8b2eda557f424aa54d6e7b3049589"
server_key_iv: "d8c7b8e592da17d27d9f3cc65fccc4e3"
server_key_version: "encryptionkey@0"
I20260923 20:22:50.813642 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "f3bade4c4eb64cf787d5f36e0e9e6999"
format_stamp: "Formatted at 2026-09-23 20:22:50 on dist-test-slave-wmd0"
server_key: "49b8b2eda557f424aa54d6e7b3049589"
server_key_iv: "d8c7b8e592da17d27d9f3cc65fccc4e3"
server_key_version: "encryptionkey@0"
I20260923 20:22:50.818868 24146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 20:22:50.822422 24160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:50.823863 24146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:50.824064 24146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "f3bade4c4eb64cf787d5f36e0e9e6999"
format_stamp: "Formatted at 2026-09-23 20:22:50 on dist-test-slave-wmd0"
server_key: "49b8b2eda557f424aa54d6e7b3049589"
server_key_iv: "d8c7b8e592da17d27d9f3cc65fccc4e3"
server_key_version: "encryptionkey@0"
I20260923 20:22:50.824323 24146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:50.840158 24146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:50.840991 24146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:50.841295 24146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:50.861040 24146 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:39889
I20260923 20:22:50.861047 24211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:39889 every 8 connection(s)
I20260923 20:22:50.862851 24146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:50.866950 24212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:50.873165 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 24146
I20260923 20:22:50.873386 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:50.874243 14847 external_mini_cluster.cc:1428] Setting key 639298c78f7dde0e807efccd992ebfa3
I20260923 20:22:50.879140 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999: Bootstrap starting.
I20260923 20:22:50.882436 24212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:50.883512 24212 log.cc:824] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:50.886943 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999: No bootstrap required, opened a new log
I20260923 20:22:50.889333 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:50.876660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.14.127.254:39889)
0923 20:22:50.877744 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:50.877756 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:50.877800 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:22:50.879685 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:50.879710 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:50.880197 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:50.880635 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:50.881257 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:50.882577 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:50.884014 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:50.884750 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:50.884951 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:50.886234 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:50.886245 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:50.886266 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:50.887040 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:50.887148 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:50.887190 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:50.887313 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:50.887584 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 20:22:50.887986 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":378,"threads_started":1}
I20260923 20:22:50.895982 24212 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bade4c4eb64cf787d5f36e0e9e6999" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39889 } }
I20260923 20:22:50.896360 24212 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:50.896436 24212 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bade4c4eb64cf787d5f36e0e9e6999, State: Initialized, Role: FOLLOWER
I20260923 20:22:50.897015 24212 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bade4c4eb64cf787d5f36e0e9e6999" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39889 } }
I20260923 20:22:50.897276 24212 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:50.897449 24212 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:50.897652 24212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:50.900077 24212 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bade4c4eb64cf787d5f36e0e9e6999" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39889 } }
I20260923 20:22:50.900573 24212 leader_election.cc:302] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3bade4c4eb64cf787d5f36e0e9e6999; no voters:
I20260923 20:22:50.901105 24212 leader_election.cc:288] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:50.901271 24217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:50.901965 24217 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [term 1 LEADER]: Becoming Leader. State: Replica: f3bade4c4eb64cf787d5f36e0e9e6999, State: Running, Role: LEADER
I20260923 20:22:50.902549 24217 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bade4c4eb64cf787d5f36e0e9e6999" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39889 } }
I20260923 20:22:50.903275 24212 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:50.905915 24218 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3bade4c4eb64cf787d5f36e0e9e6999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bade4c4eb64cf787d5f36e0e9e6999" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39889 } } }
I20260923 20:22:50.906286 24218 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:50.905979 24219 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3bade4c4eb64cf787d5f36e0e9e6999. Latest consensus state: current_term: 1 leader_uuid: "f3bade4c4eb64cf787d5f36e0e9e6999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bade4c4eb64cf787d5f36e0e9e6999" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39889 } } }
I20260923 20:22:50.906404 24219 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:50.908077 24222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:50.913754 24222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:50.923625 24222 catalog_manager.cc:1347] Generated new cluster ID: 05df7e9d9f5145f78a7947226dab70a8
I20260923 20:22:50.923801 24222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:50.934669 24222 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:50.936053 24222 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:50.945763 24222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3bade4c4eb64cf787d5f36e0e9e6999: Generated new TSK 0
I20260923 20:22:50.946708 24222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:22:50.951489 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39889
--builtin_ntp_servers=127.14.127.212:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 20:22:51.317307 24236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:51.317589 24236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:51.317750 24236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:51.325776 24236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:51.325881 24236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:51.326036 24236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:51.335528 24236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:51.336833 24236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:51.338598 24236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:51.351642 24241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:51.351692 24242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:51.351946 24244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:51.352809 24236 server_base.cc:1034] running on GCE node
I20260923 20:22:51.353442 24236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:51.354405 24236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:51.355595 24236 hybrid_clock.cc:648] HybridClock initialized: now 1790194971355527 us; error 59 us; skew 500 ppm
I20260923 20:22:51.355916 24236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:51.358053 24236 webserver.cc:460] Webserver started at http://127.14.127.193:36357/ using document root <none> and password file <none>
I20260923 20:22:51.358817 24236 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:51.358935 24236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:51.359261 24236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:51.362001 24236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "53f7b5419a7e4a578795b7c4bc76c781"
format_stamp: "Formatted at 2026-09-23 20:22:51 on dist-test-slave-wmd0"
server_key: "9999465151d6ea2849d68167a09b9e74"
server_key_iv: "6d2dd7b0a2d5ffcd4bb3273c34318cc1"
server_key_version: "encryptionkey@0"
I20260923 20:22:51.362839 24236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "53f7b5419a7e4a578795b7c4bc76c781"
format_stamp: "Formatted at 2026-09-23 20:22:51 on dist-test-slave-wmd0"
server_key: "9999465151d6ea2849d68167a09b9e74"
server_key_iv: "6d2dd7b0a2d5ffcd4bb3273c34318cc1"
server_key_version: "encryptionkey@0"
I20260923 20:22:51.367856 24236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:22:51.371258 24250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:51.372591 24236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:51.372761 24236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "53f7b5419a7e4a578795b7c4bc76c781"
format_stamp: "Formatted at 2026-09-23 20:22:51 on dist-test-slave-wmd0"
server_key: "9999465151d6ea2849d68167a09b9e74"
server_key_iv: "6d2dd7b0a2d5ffcd4bb3273c34318cc1"
server_key_version: "encryptionkey@0"
I20260923 20:22:51.372967 24236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:51.385588 24236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:51.386425 24236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:51.386746 24236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:51.387667 24236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:51.389076 24236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:51.389199 24236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:51.389319 24236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:51.389400 24236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:51.424088 24236 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:40353
I20260923 20:22:51.424139 24362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:40353 every 8 connection(s)
I20260923 20:22:51.425402 24236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:51.427886 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 24236
I20260923 20:22:51.428089 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:51.428900 14847 external_mini_cluster.cc:1428] Setting key b3b36c7b7bfcc00263fcab4d8ab1b45e
I20260923 20:22:51.431962 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39889
--builtin_ntp_servers=127.14.127.212:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 20:22:51.439354 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:51.428143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:33667 (local address 127.14.127.254:39889)
0923 20:22:51.428743 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:51.428748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:51.430233 (+ 1485us) server_negotiation.cc:407] Connection header received
0923 20:22:51.432357 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:51.432363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:51.432459 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:51.432574 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:51.433650 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:51.434696 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:51.436527 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:51.436943 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:51.437081 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:51.438485 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:51.438490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:51.438493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:51.438511 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:51.438552 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:51.438564 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:51.438691 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:51.438917 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 20:22:51.439017 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":291,"threads_started":1}
I20260923 20:22:51.440407 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:51.428992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:39889 (local address 127.14.127.193:33667)
0923 20:22:51.430054 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:51.430119 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:51.432043 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:51.432814 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:51.432822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:51.432914 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:51.433485 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:51.433502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:51.434859 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:51.434864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:51.436329 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:51.436342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:51.437180 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:51.437189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:51.437431 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:51.438152 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:51.438154 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:51.438195 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:51.438286 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:51.438691 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:51.438704 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:51.438714 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:22:51.438908 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 20:22:51.439326 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":302,"threads_started":1}
I20260923 20:22:51.442020 24363 heartbeater.cc:347] Connected to a master server at 127.14.127.254:39889
I20260923 20:22:51.442363 24363 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:51.443354 24363 heartbeater.cc:511] Master 127.14.127.254:39889 requested a full tablet report, sending...
I20260923 20:22:51.446024 24177 ts_manager.cc:194] Registered new tserver with Master: 53f7b5419a7e4a578795b7c4bc76c781 (127.14.127.193:40353)
I20260923 20:22:51.448436 24177 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.193:33667
W20260923 20:22:51.756342 24367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:51.756616 24367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:51.756719 24367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:51.763823 24367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:51.763921 24367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:51.764024 24367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:51.772840 24367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:51.774055 24367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:51.775772 24367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:51.788573 24372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:51.788619 24375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:51.788671 24373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:51.792028 24367 server_base.cc:1034] running on GCE node
I20260923 20:22:51.792685 24367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:51.793637 24367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:51.794874 24367 hybrid_clock.cc:648] HybridClock initialized: now 1790194971794811 us; error 48 us; skew 500 ppm
I20260923 20:22:51.795233 24367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:51.797523 24367 webserver.cc:460] Webserver started at http://127.14.127.194:42001/ using document root <none> and password file <none>
I20260923 20:22:51.798218 24367 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:51.798343 24367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:51.798718 24367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:51.801534 24367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "5533c9622d594ecfbb51358b7e05c197"
format_stamp: "Formatted at 2026-09-23 20:22:51 on dist-test-slave-wmd0"
server_key: "9d97a7ce85a2a4029e2b06ac26ccb867"
server_key_iv: "8f293c3f4f323fff30b1dce77a89fc01"
server_key_version: "encryptionkey@0"
I20260923 20:22:51.802325 24367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "5533c9622d594ecfbb51358b7e05c197"
format_stamp: "Formatted at 2026-09-23 20:22:51 on dist-test-slave-wmd0"
server_key: "9d97a7ce85a2a4029e2b06ac26ccb867"
server_key_iv: "8f293c3f4f323fff30b1dce77a89fc01"
server_key_version: "encryptionkey@0"
I20260923 20:22:51.807566 24367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 20:22:51.811086 24381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:51.812428 24367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:22:51.812603 24367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "5533c9622d594ecfbb51358b7e05c197"
format_stamp: "Formatted at 2026-09-23 20:22:51 on dist-test-slave-wmd0"
server_key: "9d97a7ce85a2a4029e2b06ac26ccb867"
server_key_iv: "8f293c3f4f323fff30b1dce77a89fc01"
server_key_version: "encryptionkey@0"
I20260923 20:22:51.812816 24367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:51.839246 24367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:51.840090 24367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:51.840416 24367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:51.841408 24367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:51.842957 24367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:51.843073 24367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:51.843196 24367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:51.843288 24367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:51.879578 24367 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:42449
I20260923 20:22:51.879660 24493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:42449 every 8 connection(s)
I20260923 20:22:51.880833 24367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:51.885787 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 24367
I20260923 20:22:51.886040 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:51.886539 14847 external_mini_cluster.cc:1428] Setting key b7bd8de4af888e28b4012c860ce6924d
I20260923 20:22:51.889122 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39889
--builtin_ntp_servers=127.14.127.212:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 20:22:51.894281 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:51.883477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:57473 (local address 127.14.127.254:39889)
0923 20:22:51.883650 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:51.883655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:51.885096 (+ 1441us) server_negotiation.cc:407] Connection header received
0923 20:22:51.886983 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:51.886989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:51.887080 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:51.887199 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:51.888270 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:51.888964 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:51.891396 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:51.891772 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:51.891915 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:51.893487 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:51.893491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:51.893495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:51.893514 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:51.893544 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:51.893556 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:51.893648 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:51.893877 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:22:51.893976 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:22:51.895401 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:51.884023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:39889 (local address 127.14.127.194:57473)
0923 20:22:51.884934 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:51.884974 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:51.886712 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:51.887455 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:51.887464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:51.887558 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:51.888108 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:51.888124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:51.889108 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:51.889114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:51.891239 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:51.891255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:51.892005 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:51.892012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:51.892357 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:51.893149 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:51.893151 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:51.893190 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:51.893270 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:51.893647 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:51.893658 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:51.893668 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:22:51.893860 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 20:22:51.894250 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":299,"threads_started":1}
I20260923 20:22:51.896853 24494 heartbeater.cc:347] Connected to a master server at 127.14.127.254:39889
I20260923 20:22:51.897157 24494 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:51.898030 24494 heartbeater.cc:511] Master 127.14.127.254:39889 requested a full tablet report, sending...
I20260923 20:22:51.899531 24177 ts_manager.cc:194] Registered new tserver with Master: 5533c9622d594ecfbb51358b7e05c197 (127.14.127.194:42449)
I20260923 20:22:51.900329 24177 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.194:57473
W20260923 20:22:52.225677 24497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:52.225996 24497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:52.226151 24497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:52.234243 24497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:52.234364 24497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:52.234535 24497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:52.244045 24497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:52.245314 24497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:52.247483 24497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:52.260804 24505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:52.260869 24503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:52.260902 24502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:52.261628 24497 server_base.cc:1034] running on GCE node
I20260923 20:22:52.262188 24497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:52.263146 24497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:52.264370 24497 hybrid_clock.cc:648] HybridClock initialized: now 1790194972264332 us; error 54 us; skew 500 ppm
I20260923 20:22:52.264693 24497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:52.267064 24497 webserver.cc:460] Webserver started at http://127.14.127.195:33647/ using document root <none> and password file <none>
I20260923 20:22:52.267951 24497 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:52.268095 24497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:52.268420 24497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:52.271335 24497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "a266ea3655d545b09c4674a06837e19c"
format_stamp: "Formatted at 2026-09-23 20:22:52 on dist-test-slave-wmd0"
server_key: "12e018c64fa9eb237178f384038d5f91"
server_key_iv: "2126e54856cf5a9160cfe52f3a930ed5"
server_key_version: "encryptionkey@0"
I20260923 20:22:52.272161 24497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "a266ea3655d545b09c4674a06837e19c"
format_stamp: "Formatted at 2026-09-23 20:22:52 on dist-test-slave-wmd0"
server_key: "12e018c64fa9eb237178f384038d5f91"
server_key_iv: "2126e54856cf5a9160cfe52f3a930ed5"
server_key_version: "encryptionkey@0"
I20260923 20:22:52.277519 24497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:22:52.281200 24511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:52.282546 24497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:22:52.282764 24497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "a266ea3655d545b09c4674a06837e19c"
format_stamp: "Formatted at 2026-09-23 20:22:52 on dist-test-slave-wmd0"
server_key: "12e018c64fa9eb237178f384038d5f91"
server_key_iv: "2126e54856cf5a9160cfe52f3a930ed5"
server_key_version: "encryptionkey@0"
I20260923 20:22:52.282984 24497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:52.310730 24497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:52.311650 24497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:52.311949 24497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:52.313007 24497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:52.314440 24497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:52.314561 24497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:52.314705 24497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:52.314800 24497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:52.347965 24497 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:35759
I20260923 20:22:52.348049 24623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:35759 every 8 connection(s)
I20260923 20:22:52.349251 24497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:52.353770 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 24497
I20260923 20:22:52.353924 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:52.354389 14847 external_mini_cluster.cc:1428] Setting key 38ca32ec6583c1095b52d9ae29a775bb
I20260923 20:22:52.361552 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:52.351709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:35033 (local address 127.14.127.254:39889)
0923 20:22:52.351916 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:52.351921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:52.353283 (+ 1362us) server_negotiation.cc:407] Connection header received
0923 20:22:52.355330 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:52.355335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:52.355427 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:52.355549 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:52.356603 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.357345 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.358922 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.359331 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.359458 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:52.360747 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:52.360752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:52.360755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:52.360772 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:52.360815 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:52.360828 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:52.360923 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:52.361135 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 20:22:52.361231 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:22:52.362572 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:52.352222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:39889 (local address 127.14.127.195:35033)
0923 20:22:52.353125 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:52.353169 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:52.355007 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:52.355782 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:52.355791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:52.355886 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:52.356440 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:52.356457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.357491 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.357501 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:52.358785 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:52.358797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.359557 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.359564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:52.359793 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:52.360432 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:52.360434 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:52.360471 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:52.360551 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:52.360921 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:52.360933 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:52.360943 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:22:52.361129 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 20:22:52.361522 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":260,"threads_started":1}
I20260923 20:22:52.363878 24624 heartbeater.cc:347] Connected to a master server at 127.14.127.254:39889
I20260923 20:22:52.364130 24624 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:52.364805 24624 heartbeater.cc:511] Master 127.14.127.254:39889 requested a full tablet report, sending...
I20260923 20:22:52.366052 24177 ts_manager.cc:194] Registered new tserver with Master: a266ea3655d545b09c4674a06837e19c (127.14.127.195:35759)
I20260923 20:22:52.366932 24177 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.195:35033
I20260923 20:22:52.369483 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:22:52.375681 24365 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:52.374096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.14.127.254:39889)
0923 20:22:52.374303 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:52.374308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:52.374799 (+ 491us) server_negotiation.cc:407] Connection header received
0923 20:22:52.374930 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:52.374933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:52.375009 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:52.375094 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:52.375500 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42462: BlockingRecv error: recv got EOF from 127.0.0.1:42462 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:22:52.376616 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 24236
I20260923 20:22:52.384438 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:52.377561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.14.127.193:40353)
0923 20:22:52.378117 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:52.378124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:52.378150 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:22:52.378209 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:52.378215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:52.378415 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:52.378539 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:52.378934 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.379908 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.381170 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.381721 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.381830 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:52.382949 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:52.382957 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:52.382970 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:52.383684 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:52.383750 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:52.383796 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:52.383919 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:52.384020 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 20:22:52.384149 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":315,"threads_started":1}
I20260923 20:22:52.385638 24358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:52.466991 24363 heartbeater.cc:503] Master 127.14.127.254:39889 was elected leader, sending a full tablet report...
I20260923 20:22:52.468456 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 24236
W20260923 20:22:52.485361 24161 connection.cc:535] server connection from 127.14.127.193:33667 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 20:22:52.485654 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 24367
I20260923 20:22:52.493587 24635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:52.486530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.14.127.194:42449)
0923 20:22:52.487145 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:52.487151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:52.487186 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:22:52.487274 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:52.487284 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:52.487562 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:52.487745 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:52.488153 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.489143 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.490378 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.491025 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.491164 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:52.492183 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:52.492192 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:52.492205 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:52.492792 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:52.492840 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:52.492861 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:52.493000 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:52.493119 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 20:22:52.493245 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":329,"threads_started":1}
I20260923 20:22:52.494860 24489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:52.575148 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 24367
I20260923 20:22:52.591367 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 24497
I20260923 20:22:52.599270 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:52.592362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.14.127.195:35759)
0923 20:22:52.593040 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:52.593047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:52.593077 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:22:52.593135 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:52.593141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:52.593353 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:52.593482 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:52.593874 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.594992 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.596226 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:52.596762 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:52.596853 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:52.597814 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:52.597822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:52.597835 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:52.598431 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:52.598475 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:52.598495 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:52.598594 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:52.598793 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 20:22:52.598929 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":365,"threads_started":1}
I20260923 20:22:52.600481 24619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:52.681519 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 24497
I20260923 20:22:52.697633 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 24146
I20260923 20:22:52.698966 24207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:52.800845 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 24146
2026-09-23T20:22:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2429 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T20:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:52Z Disabled control of system clock
I20260923 20:22:52.870594 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:44501
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:44501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:22:53.190220 24643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:53.190507 24643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:53.190692 24643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:53.198024 24643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:53.198127 24643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:53.198195 24643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:53.198324 24643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:53.198400 24643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:53.207681 24643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:44501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:44501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:53.209016 24643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:53.210830 24643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:53.222452 24649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:53.222568 24651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:53.222488 24648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:53.224377 24643 server_base.cc:1034] running on GCE node
I20260923 20:22:53.225129 24643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:53.226526 24643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:53.227782 24643 hybrid_clock.cc:648] HybridClock initialized: now 1790194973227708 us; error 63 us; skew 500 ppm
I20260923 20:22:53.228153 24643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:53.230376 24643 webserver.cc:460] Webserver started at http://127.14.127.254:38077/ using document root <none> and password file <none>
I20260923 20:22:53.231108 24643 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:53.231243 24643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:53.231568 24643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:53.234405 24643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "740f20fab8a0447c92b70d11bfd5c435"
format_stamp: "Formatted at 2026-09-23 20:22:53 on dist-test-slave-wmd0"
server_key: "11a4bee5259bc0c6699c97f1ea99aaad"
server_key_iv: "e0f43c4e67dad2e2aa6f79924ec1fcb1"
server_key_version: "encryptionkey@0"
I20260923 20:22:53.235263 24643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "740f20fab8a0447c92b70d11bfd5c435"
format_stamp: "Formatted at 2026-09-23 20:22:53 on dist-test-slave-wmd0"
server_key: "11a4bee5259bc0c6699c97f1ea99aaad"
server_key_iv: "e0f43c4e67dad2e2aa6f79924ec1fcb1"
server_key_version: "encryptionkey@0"
I20260923 20:22:53.240334 24643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:22:53.243690 24657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:53.245002 24643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:53.245175 24643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "740f20fab8a0447c92b70d11bfd5c435"
format_stamp: "Formatted at 2026-09-23 20:22:53 on dist-test-slave-wmd0"
server_key: "11a4bee5259bc0c6699c97f1ea99aaad"
server_key_iv: "e0f43c4e67dad2e2aa6f79924ec1fcb1"
server_key_version: "encryptionkey@0"
I20260923 20:22:53.245394 24643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:53.251343 24643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:53.252096 24643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:53.252399 24643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:53.271054 24643 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:44501
I20260923 20:22:53.271138 24708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:44501 every 8 connection(s)
I20260923 20:22:53.272434 24643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:53.276506 24709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:53.280588 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 24643
I20260923 20:22:53.280814 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:53.281612 14847 external_mini_cluster.cc:1428] Setting key 3b8e94cf0fb1eaec43b6bddbc0b38087
I20260923 20:22:53.288470 24709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435: Bootstrap starting.
I20260923 20:22:53.290251 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:53.283739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.14.127.254:44501)
0923 20:22:53.284712 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:53.284729 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:53.284775 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:22:53.286912 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:53.286934 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:53.287356 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:53.287665 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:53.288727 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:53.288733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:53.288749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:53.288772 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:53.288830 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:53.288845 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:53.289004 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:53.289125 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 20:22:53.289398 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":366,"threads_started":1}
I20260923 20:22:53.291937 24709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:53.293006 24709 log.cc:824] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:53.295681 24709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435: No bootstrap required, opened a new log
I20260923 20:22:53.302884 24709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f20fab8a0447c92b70d11bfd5c435" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44501 } }
I20260923 20:22:53.303212 24709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:53.303288 24709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740f20fab8a0447c92b70d11bfd5c435, State: Initialized, Role: FOLLOWER
I20260923 20:22:53.303819 24709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f20fab8a0447c92b70d11bfd5c435" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44501 } }
I20260923 20:22:53.304009 24709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:53.304104 24709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:53.304231 24709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:53.306602 24709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f20fab8a0447c92b70d11bfd5c435" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44501 } }
I20260923 20:22:53.307114 24709 leader_election.cc:302] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 740f20fab8a0447c92b70d11bfd5c435; no voters:
I20260923 20:22:53.307670 24709 leader_election.cc:288] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:53.307848 24714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:53.308629 24714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [term 1 LEADER]: Becoming Leader. State: Replica: 740f20fab8a0447c92b70d11bfd5c435, State: Running, Role: LEADER
I20260923 20:22:53.309123 24714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f20fab8a0447c92b70d11bfd5c435" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44501 } }
I20260923 20:22:53.309823 24709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:53.311894 24716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 740f20fab8a0447c92b70d11bfd5c435. Latest consensus state: current_term: 1 leader_uuid: "740f20fab8a0447c92b70d11bfd5c435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f20fab8a0447c92b70d11bfd5c435" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44501 } } }
I20260923 20:22:53.312358 24716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:53.313158 24715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "740f20fab8a0447c92b70d11bfd5c435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f20fab8a0447c92b70d11bfd5c435" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44501 } } }
I20260923 20:22:53.313598 24715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:53.315049 24721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:53.320315 24721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:53.329434 24721 catalog_manager.cc:1347] Generated new cluster ID: 75fee3dd95224fb5827e29259f06207b
I20260923 20:22:53.329560 24721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:53.349303 24721 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:53.349828 24721 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:53.363538 24721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 740f20fab8a0447c92b70d11bfd5c435: Generated new TSK 0
I20260923 20:22:53.364382 24721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:22:53.375497 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44501
--builtin_ntp_servers=127.14.127.212:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:22:53.727624 24733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:53.727911 24733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:53.728060 24733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:53.735229 24733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:53.735327 24733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:53.735492 24733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:53.744403 24733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:53.745738 24733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:53.747565 24733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:53.760941 24739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:53.760895 24741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:53.760985 24738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:53.761646 24733 server_base.cc:1034] running on GCE node
I20260923 20:22:53.762279 24733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:53.763312 24733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:53.764518 24733 hybrid_clock.cc:648] HybridClock initialized: now 1790194973764448 us; error 57 us; skew 500 ppm
I20260923 20:22:53.764856 24733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:53.767359 24733 webserver.cc:460] Webserver started at http://127.14.127.193:38023/ using document root <none> and password file <none>
I20260923 20:22:53.768074 24733 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:53.768184 24733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:53.768514 24733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:53.773245 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "276843c997824a119f4e43b4627c409c"
format_stamp: "Formatted at 2026-09-23 20:22:53 on dist-test-slave-wmd0"
server_key: "e23566c38e2e3a9b377addbfba09ded3"
server_key_iv: "b83bda4cbd60c26686368f48b77d4b5e"
server_key_version: "encryptionkey@0"
I20260923 20:22:53.774063 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "276843c997824a119f4e43b4627c409c"
format_stamp: "Formatted at 2026-09-23 20:22:53 on dist-test-slave-wmd0"
server_key: "e23566c38e2e3a9b377addbfba09ded3"
server_key_iv: "b83bda4cbd60c26686368f48b77d4b5e"
server_key_version: "encryptionkey@0"
I20260923 20:22:53.779434 24733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:22:53.782864 24747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:53.784145 24733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:22:53.784348 24733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "276843c997824a119f4e43b4627c409c"
format_stamp: "Formatted at 2026-09-23 20:22:53 on dist-test-slave-wmd0"
server_key: "e23566c38e2e3a9b377addbfba09ded3"
server_key_iv: "b83bda4cbd60c26686368f48b77d4b5e"
server_key_version: "encryptionkey@0"
I20260923 20:22:53.784567 24733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:53.790179 24733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:53.791034 24733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:53.791306 24733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:53.792189 24733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:53.793612 24733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:53.793730 24733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:53.793852 24733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:53.793948 24733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:53.826349 24733 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:39947
I20260923 20:22:53.826431 24859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:39947 every 8 connection(s)
I20260923 20:22:53.827680 24733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:53.829808 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 24733
I20260923 20:22:53.830039 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:53.830864 14847 external_mini_cluster.cc:1428] Setting key c81f4ce9a40410b11d50f7959023f4f9
I20260923 20:22:53.833914 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44501
--builtin_ntp_servers=127.14.127.212:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 20:22:53.836951 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:53.830290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:33521 (local address 127.14.127.254:44501)
0923 20:22:53.830821 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:53.830827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:53.832201 (+ 1374us) server_negotiation.cc:407] Connection header received
0923 20:22:53.834519 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:53.834523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:53.834576 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:53.834727 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:53.836252 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:53.836256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:53.836259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:53.836276 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:53.836324 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:53.836336 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:53.836444 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:53.836662 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 20:22:53.836715 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":232,"threads_started":1}
I20260923 20:22:53.838193 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:53.830940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:44501 (local address 127.14.127.193:33521)
0923 20:22:53.832064 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:53.832099 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:53.834235 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:53.834993 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:53.835008 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:53.835138 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:53.835864 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:53.835870 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:53.835927 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:53.836047 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:53.836443 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:53.836456 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:53.836466 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:22:53.836734 (+ 268us) client_negotiation.cc:240] Negotiation successful
0923 20:22:53.836996 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":365,"threads_started":1}
I20260923 20:22:53.839805 24860 heartbeater.cc:347] Connected to a master server at 127.14.127.254:44501
I20260923 20:22:53.840118 24860 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:53.840761 24860 heartbeater.cc:511] Master 127.14.127.254:44501 requested a full tablet report, sending...
I20260923 20:22:53.842926 24674 ts_manager.cc:194] Registered new tserver with Master: 276843c997824a119f4e43b4627c409c (127.14.127.193:39947)
W20260923 20:22:54.158866 24864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:54.159133 24864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:54.159274 24864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:54.166385 24864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:54.166484 24864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:54.166709 24864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:54.176095 24864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:54.177419 24864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:54.179216 24864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:54.192031 24870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:54.192114 24869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:54.192121 24872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:54.192816 24864 server_base.cc:1034] running on GCE node
I20260923 20:22:54.193414 24864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:54.194334 24864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:54.195559 24864 hybrid_clock.cc:648] HybridClock initialized: now 1790194974195515 us; error 53 us; skew 500 ppm
I20260923 20:22:54.195920 24864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:54.198220 24864 webserver.cc:460] Webserver started at http://127.14.127.194:41721/ using document root <none> and password file <none>
I20260923 20:22:54.199007 24864 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:54.199134 24864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:54.199455 24864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:54.202302 24864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "63fbf3499dd94191b00c5f8f50777672"
format_stamp: "Formatted at 2026-09-23 20:22:54 on dist-test-slave-wmd0"
server_key: "9f414a8f37414a426caee678a75ffae7"
server_key_iv: "e1587c3a64589e6ec6b46096b8189576"
server_key_version: "encryptionkey@0"
I20260923 20:22:54.203111 24864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "63fbf3499dd94191b00c5f8f50777672"
format_stamp: "Formatted at 2026-09-23 20:22:54 on dist-test-slave-wmd0"
server_key: "9f414a8f37414a426caee678a75ffae7"
server_key_iv: "e1587c3a64589e6ec6b46096b8189576"
server_key_version: "encryptionkey@0"
I20260923 20:22:54.208374 24864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:22:54.211845 24878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:54.213081 24864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:22:54.213248 24864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "63fbf3499dd94191b00c5f8f50777672"
format_stamp: "Formatted at 2026-09-23 20:22:54 on dist-test-slave-wmd0"
server_key: "9f414a8f37414a426caee678a75ffae7"
server_key_iv: "e1587c3a64589e6ec6b46096b8189576"
server_key_version: "encryptionkey@0"
I20260923 20:22:54.213459 24864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:54.219475 24864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:54.220257 24864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:54.220521 24864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:54.221439 24864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:54.222942 24864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:54.223059 24864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:54.223187 24864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:54.223268 24864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:54.256579 24864 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:40581
I20260923 20:22:54.256613 24990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:40581 every 8 connection(s)
I20260923 20:22:54.257943 24864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:54.266108 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:54.260496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:51971 (local address 127.14.127.254:44501)
0923 20:22:54.260694 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:54.260699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:54.262139 (+ 1440us) server_negotiation.cc:407] Connection header received
0923 20:22:54.263877 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:54.263882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:54.263952 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:54.264073 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:54.265353 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:54.265359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:54.265363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:54.265383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:54.265421 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:54.265432 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:54.265555 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:54.265762 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 20:22:54.265834 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:22:54.267037 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:54.261022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:44501 (local address 127.14.127.194:51971)
0923 20:22:54.261983 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:54.262017 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:54.263599 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:54.264307 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:54.264315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:54.264409 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:54.265011 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:54.265013 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:54.265049 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:54.265133 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:54.265525 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:54.265536 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:54.265542 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:22:54.265837 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 20:22:54.266094 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":318,"threads_started":1}
I20260923 20:22:54.267570 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 24864
I20260923 20:22:54.267786 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:54.268354 14847 external_mini_cluster.cc:1428] Setting key b56b60a51d6b60684684cc528d75d0cd
I20260923 20:22:54.268594 24991 heartbeater.cc:347] Connected to a master server at 127.14.127.254:44501
I20260923 20:22:54.268914 24991 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:54.269632 24991 heartbeater.cc:511] Master 127.14.127.254:44501 requested a full tablet report, sending...
I20260923 20:22:54.271098 24674 ts_manager.cc:194] Registered new tserver with Master: 63fbf3499dd94191b00c5f8f50777672 (127.14.127.194:40581)
I20260923 20:22:54.271406 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44501
--builtin_ntp_servers=127.14.127.212:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:22:54.599951 24994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:54.600265 24994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:54.600419 24994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:54.607614 24994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:54.607739 24994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:54.607914 24994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:54.617159 24994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:54.618551 24994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:54.620461 24994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:54.633224 25000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:54.633320 25002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:54.633294 24999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:54.634317 24994 server_base.cc:1034] running on GCE node
I20260923 20:22:54.634946 24994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:54.635890 24994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:54.637118 24994 hybrid_clock.cc:648] HybridClock initialized: now 1790194974637072 us; error 41 us; skew 500 ppm
I20260923 20:22:54.637451 24994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:22:54.639690 24994 webserver.cc:460] Webserver started at http://127.14.127.195:36111/ using document root <none> and password file <none>
I20260923 20:22:54.640388 24994 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:54.640511 24994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:54.640825 24994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:54.643646 24994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "05da2ae7764c4e44bade8d2f5f418ae2"
format_stamp: "Formatted at 2026-09-23 20:22:54 on dist-test-slave-wmd0"
server_key: "e7784493442425d2c3abeffa4fc557ab"
server_key_iv: "62f580bdd9693320d3cd264776d23880"
server_key_version: "encryptionkey@0"
I20260923 20:22:54.644424 24994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "05da2ae7764c4e44bade8d2f5f418ae2"
format_stamp: "Formatted at 2026-09-23 20:22:54 on dist-test-slave-wmd0"
server_key: "e7784493442425d2c3abeffa4fc557ab"
server_key_iv: "62f580bdd9693320d3cd264776d23880"
server_key_version: "encryptionkey@0"
I20260923 20:22:54.649647 24994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:22:54.653290 25008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:54.654597 24994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:22:54.654834 24994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "05da2ae7764c4e44bade8d2f5f418ae2"
format_stamp: "Formatted at 2026-09-23 20:22:54 on dist-test-slave-wmd0"
server_key: "e7784493442425d2c3abeffa4fc557ab"
server_key_iv: "62f580bdd9693320d3cd264776d23880"
server_key_version: "encryptionkey@0"
I20260923 20:22:54.655061 24994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:54.660780 24994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:54.661557 24994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:54.661836 24994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:54.662797 24994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:54.664206 24994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:54.664327 24994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:54.664461 24994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:54.664546 24994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:54.698161 24994 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:43943
I20260923 20:22:54.698210 25120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:43943 every 8 connection(s)
I20260923 20:22:54.699477 24994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:54.707234 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 24994
I20260923 20:22:54.707427 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:54.707442 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:54.701989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:56375 (local address 127.14.127.254:44501)
0923 20:22:54.702311 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:54.702317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:54.703703 (+ 1386us) server_negotiation.cc:407] Connection header received
0923 20:22:54.705303 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:54.705307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:54.705367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:54.705457 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:54.706682 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:54.706686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:54.706689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:54.706707 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:54.706737 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:54.706748 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:54.706886 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:54.707079 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 20:22:54.707150 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260923 20:22:54.707891 14847 external_mini_cluster.cc:1428] Setting key cd526eb96e0e0ff8e981c5d065ef7d81
I20260923 20:22:54.708380 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:54.702652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:44501 (local address 127.14.127.195:56375)
0923 20:22:54.703536 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:54.703573 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:54.705061 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:54.705647 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:54.705655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:54.705752 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:54.706354 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:22:54.706356 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:54.706398 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:22:54.706479 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:54.706856 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:54.706867 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:54.706875 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:22:54.707155 (+ 280us) client_negotiation.cc:240] Negotiation successful
0923 20:22:54.707442 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":250,"threads_started":1}
I20260923 20:22:54.709720 25121 heartbeater.cc:347] Connected to a master server at 127.14.127.254:44501
I20260923 20:22:54.710040 25121 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:54.710778 25121 heartbeater.cc:511] Master 127.14.127.254:44501 requested a full tablet report, sending...
I20260923 20:22:54.712128 24674 ts_manager.cc:194] Registered new tserver with Master: 05da2ae7764c4e44bade8d2f5f418ae2 (127.14.127.195:43943)
I20260923 20:22:54.723460 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:22:54.729339 24862 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:22:54.727866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.14.127.254:44501)
0923 20:22:54.728021 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:54.728026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:54.728486 (+ 460us) server_negotiation.cc:407] Connection header received
0923 20:22:54.728612 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:54.728616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:54.728671 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:54.728763 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:54.729103 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41424: BlockingRecv error: recv got EOF from 127.0.0.1:41424 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 20:22:54.729957 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 24733
I20260923 20:22:54.733429 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:54.730935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.14.127.193:39947)
0923 20:22:54.731466 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:54.731473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:54.731498 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:22:54.731553 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:54.731559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:54.731777 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:54.731899 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:54.732747 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:54.732753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:54.732766 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:54.732785 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:54.732827 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:54.732841 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:54.732941 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:54.733016 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 20:22:54.733122 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":257,"threads_started":1}
I20260923 20:22:54.734592 24855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:54.814752 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 24733
I20260923 20:22:54.831104 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 24864
I20260923 20:22:54.834532 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:54.832025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.14.127.194:40581)
0923 20:22:54.832598 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:54.832605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:54.832632 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:22:54.832693 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:54.832699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:54.832913 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:54.833038 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:54.833872 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:54.833878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:54.833889 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:54.833900 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:54.833940 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:54.833956 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:54.834057 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:54.834140 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 20:22:54.834279 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":309,"threads_started":1}
I20260923 20:22:54.835822 24986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:54.916014 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 24864
I20260923 20:22:54.932451 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 24994
I20260923 20:22:54.939857 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:54.933350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.14.127.195:43943)
0923 20:22:54.933988 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:54.933995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:54.934024 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:22:54.934085 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:54.934091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:54.934312 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:54.934432 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:54.939040 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:54.939048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:54.939062 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:22:54.939084 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:54.939133 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:22:54.939148 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:54.939333 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:54.939409 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 20:22:54.939517 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":339,"threads_started":1}
I20260923 20:22:54.941105 25116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:55.021371 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 24994
I20260923 20:22:55.036922 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 24643
I20260923 20:22:55.038138 24704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:55.141021 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 24643
2026-09-23T20:22:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2339 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:22:55 dist-test-slave-wmd0 krb5kdc[25139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:22:55 dist-test-slave-wmd0 krb5kdc[25139](info): set up 2 sockets
Sep 23 20:22:55 dist-test-slave-wmd0 krb5kdc[25139](info): commencing operation
krb5kdc: starting...
W20260923 20:22:56.239658 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:22:56 dist-test-slave-wmd0 krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:22:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:22:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:22:56.470695 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:39365
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:39365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:56.804677 25152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:56.804982 25152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:56.805147 25152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:56.812564 25152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:22:56.812673 25152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:56.812747 25152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:56.812848 25152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:22:56.812922 25152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:22:56.822960 25152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:39365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:39365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:56.824276 25152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:56.825573 25152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:56.837328 25160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:56.837366 25158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:56.837244 25157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:56.838380 25152 server_base.cc:1034] running on GCE node
I20260923 20:22:56.839135 25152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:56.840474 25152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:56.841710 25152 hybrid_clock.cc:648] HybridClock initialized: now 1790194976841656 us; error 39 us; skew 500 ppm
I20260923 20:22:56.842065 25152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:56 dist-test-slave-wmd0 krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194976, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:56.845387 25152 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:22:56.847069 25152 webserver.cc:460] Webserver started at http://127.14.127.254:34941/ using document root <none> and password file <none>
I20260923 20:22:56.847776 25152 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:56.847913 25152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:56.848243 25152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:56.851323 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "2bbd1d5ec18d4519b017220c557be516"
format_stamp: "Formatted at 2026-09-23 20:22:56 on dist-test-slave-wmd0"
server_key: "efafe90d631f0bb40ad1d5c8b4d9721f"
server_key_iv: "b209afced5451a4a1f0bef65c034f04b"
server_key_version: "encryptionkey@0"
I20260923 20:22:56.852098 25152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "2bbd1d5ec18d4519b017220c557be516"
format_stamp: "Formatted at 2026-09-23 20:22:56 on dist-test-slave-wmd0"
server_key: "efafe90d631f0bb40ad1d5c8b4d9721f"
server_key_iv: "b209afced5451a4a1f0bef65c034f04b"
server_key_version: "encryptionkey@0"
I20260923 20:22:56.857578 25152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:22:56.861424 25167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:56.862946 25152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:56.863131 25152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "2bbd1d5ec18d4519b017220c557be516"
format_stamp: "Formatted at 2026-09-23 20:22:56 on dist-test-slave-wmd0"
server_key: "efafe90d631f0bb40ad1d5c8b4d9721f"
server_key_iv: "b209afced5451a4a1f0bef65c034f04b"
server_key_version: "encryptionkey@0"
I20260923 20:22:56.863334 25152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:56.884899 25152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:56.889277 25152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:56.889565 25152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:56.908249 25152 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:39365
I20260923 20:22:56.908298 25218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:39365 every 8 connection(s)
I20260923 20:22:56.910148 25152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:22:56.912930 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 25152
I20260923 20:22:56.913182 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:22:56.913990 14847 external_mini_cluster.cc:1428] Setting key c585c3274935219e20fbffe29ef35835
I20260923 20:22:56.914896 25219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:22:56 dist-test-slave-wmd0 krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:56.929687 25219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516: Bootstrap starting.
I20260923 20:22:56.933965 25219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:56.935272 25219 log.cc:824] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:56.938539 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:56.916078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.14.127.254:39365)
0923 20:22:56.916884 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:56.916900 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:56.916940 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:22:56.918435 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:56.918455 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:56.918916 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:56.919256 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:56.921121 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:56.922503 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:56.924292 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:56.924942 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:56.925077 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:56.929751 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:56.929764 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:56.929785 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:56.929837 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:56.933867 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:56.934642 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:56.934651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:56.934658 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:56.934862 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:56.935202 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:56.935207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:56.935213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:56.935972 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:56.936232 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:56.936773 (+ 541us) server_negotiation.cc:299] Negotiation successful
0923 20:22:56.937198 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":387,"threads_started":1}
I20260923 20:22:56.938789 25219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516: No bootstrap required, opened a new log
I20260923 20:22:56.946709 25219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd1d5ec18d4519b017220c557be516" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39365 } }
I20260923 20:22:56.947038 25219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:56.947111 25219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbd1d5ec18d4519b017220c557be516, State: Initialized, Role: FOLLOWER
I20260923 20:22:56.947599 25219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd1d5ec18d4519b017220c557be516" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39365 } }
I20260923 20:22:56.947798 25219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:22:56.947893 25219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:22:56.948016 25219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:56.950035 25219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd1d5ec18d4519b017220c557be516" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39365 } }
I20260923 20:22:56.950594 25219 leader_election.cc:302] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bbd1d5ec18d4519b017220c557be516; no voters:
I20260923 20:22:56.951191 25219 leader_election.cc:288] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:22:56.951320 25224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:56.952004 25224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [term 1 LEADER]: Becoming Leader. State: Replica: 2bbd1d5ec18d4519b017220c557be516, State: Running, Role: LEADER
I20260923 20:22:56.952528 25224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd1d5ec18d4519b017220c557be516" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39365 } }
I20260923 20:22:56.953233 25219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:22:56.955296 25226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbd1d5ec18d4519b017220c557be516. Latest consensus state: current_term: 1 leader_uuid: "2bbd1d5ec18d4519b017220c557be516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd1d5ec18d4519b017220c557be516" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39365 } } }
I20260923 20:22:56.955719 25226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [sys.catalog]: This master's current role is: LEADER
I20260923 20:22:56.956305 25225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bbd1d5ec18d4519b017220c557be516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbd1d5ec18d4519b017220c557be516" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39365 } } }
I20260923 20:22:56.956681 25225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516 [sys.catalog]: This master's current role is: LEADER
W20260923 20:22:56.966204 25240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:22:56.966313 25240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:22:56.966475 25232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:22:56.968227 25232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:22:56.975986 25232 catalog_manager.cc:1347] Generated new cluster ID: ecb161ba833f4426a90b9e282cf2f1b3
I20260923 20:22:56.976092 25232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:22:56.984014 25232 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:22:56.985738 25232 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:22:57.008352 25232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516: Generated new TSK 0
I20260923 20:22:57.009450 25232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:22:57.066330 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39365
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:57.391316 25245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:57.391594 25245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:57.391875 25245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:57.399290 25245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:57.399401 25245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:57.399569 25245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:22:57.409252 25245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:57.410570 25245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:57.411903 25245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:57.424762 25250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:57.424809 25251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:57.424728 25253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:57.425997 25245 server_base.cc:1034] running on GCE node
I20260923 20:22:57.426738 25245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:57.427726 25245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:57.428941 25245 hybrid_clock.cc:648] HybridClock initialized: now 1790194977428896 us; error 31 us; skew 500 ppm
I20260923 20:22:57.429231 25245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:57 dist-test-slave-wmd0 krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194977, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:57.432639 25245 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:22:57.434183 25245 webserver.cc:460] Webserver started at http://127.14.127.193:44707/ using document root <none> and password file <none>
I20260923 20:22:57.434968 25245 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:57.435081 25245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:57.435375 25245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:57.439078 25245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "02aa0694e3c34494ba06811f08b2ccc2"
format_stamp: "Formatted at 2026-09-23 20:22:57 on dist-test-slave-wmd0"
server_key: "50888dfb846597c63e1630accb782c1f"
server_key_iv: "ae37b22ea1985e3ea5f8ea93c95be79c"
server_key_version: "encryptionkey@0"
I20260923 20:22:57.439867 25245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "02aa0694e3c34494ba06811f08b2ccc2"
format_stamp: "Formatted at 2026-09-23 20:22:57 on dist-test-slave-wmd0"
server_key: "50888dfb846597c63e1630accb782c1f"
server_key_iv: "ae37b22ea1985e3ea5f8ea93c95be79c"
server_key_version: "encryptionkey@0"
I20260923 20:22:57.445124 25245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:22:57.448761 25260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:57.450209 25245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:22:57.450371 25245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "02aa0694e3c34494ba06811f08b2ccc2"
format_stamp: "Formatted at 2026-09-23 20:22:57 on dist-test-slave-wmd0"
server_key: "50888dfb846597c63e1630accb782c1f"
server_key_iv: "ae37b22ea1985e3ea5f8ea93c95be79c"
server_key_version: "encryptionkey@0"
I20260923 20:22:57.450528 25245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:57.477820 25245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:57.482199 25245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:57.482450 25245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:57.483399 25245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:57.484661 25245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:57.484735 25245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:57.484819 25245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:57.484866 25245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:57.517266 25245 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:33845
I20260923 20:22:57.517279 25372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:33845 every 8 connection(s)
I20260923 20:22:57.518573 25245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:22:57.527823 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 25245
I20260923 20:22:57.528095 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:22:57.528889 14847 external_mini_cluster.cc:1428] Setting key 7aa2a7d1ae4fbdec143c1a86e1520635
Sep 23 20:22:57 dist-test-slave-wmd0 krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194977, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:57.540280 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:57.521165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:35267 (local address 127.14.127.254:39365)
0923 20:22:57.521785 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:57.521789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:57.522843 (+ 1054us) server_negotiation.cc:407] Connection header received
0923 20:22:57.524663 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:57.524669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:57.524762 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:57.524886 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:57.526377 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:57.527166 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:57.529232 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:57.529859 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:57.530002 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:57.534039 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:57.534044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:57.534047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:57.534081 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:57.536579 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:57.537585 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:57.537589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:57.537591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:57.537707 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:57.538221 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:57.538229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:57.538233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:57.538642 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:57.538800 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:57.539508 (+ 708us) server_negotiation.cc:299] Negotiation successful
0923 20:22:57.539788 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260923 20:22:57.540895 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:57.521948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:39365 (local address 127.14.127.193:35267)
0923 20:22:57.522699 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:57.522738 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:57.524393 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:57.525129 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:57.525138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:57.525651 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:57.526202 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:57.526219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:57.527326 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:57.527330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:57.529075 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:57.529094 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:57.530090 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:57.530099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:57.530341 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:57.531073 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:57.531096 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:57.533804 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:57.536744 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:57.536757 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:57.536778 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:57.537456 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:57.537819 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:57.537826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:57.537833 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:57.538101 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:57.538767 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:57.538773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:57.539116 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 20:22:57.539355 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 20:22:57.539761 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":275,"threads_started":1}
I20260923 20:22:57.542495 25373 heartbeater.cc:347] Connected to a master server at 127.14.127.254:39365
I20260923 20:22:57.542871 25373 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:57.543713 25373 heartbeater.cc:511] Master 127.14.127.254:39365 requested a full tablet report, sending...
I20260923 20:22:57.545964 25184 ts_manager.cc:194] Registered new tserver with Master: 02aa0694e3c34494ba06811f08b2ccc2 (127.14.127.193:33845)
I20260923 20:22:57.547499 25184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:35267
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:22:57.581449 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39365
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:57.902241 25379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:57.902577 25379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:57.902773 25379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:57.910668 25379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:57.910820 25379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:57.911037 25379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:22:57.925289 25379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:57.926651 25379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:57.927997 25379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:57.941361 25384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:57.941251 25385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:57.941406 25387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:57.947502 25379 server_base.cc:1034] running on GCE node
I20260923 20:22:57.948261 25379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:57.949306 25379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:57.950546 25379 hybrid_clock.cc:648] HybridClock initialized: now 1790194977950496 us; error 49 us; skew 500 ppm
I20260923 20:22:57.950903 25379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:57 dist-test-slave-wmd0 krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194977, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:57.954110 25379 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:22:57.955899 25379 webserver.cc:460] Webserver started at http://127.14.127.194:44471/ using document root <none> and password file <none>
I20260923 20:22:57.956641 25379 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:57.956784 25379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:57.957087 25379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:57.960382 25379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "83b558d2a1d541adb43b5927df9b34f4"
format_stamp: "Formatted at 2026-09-23 20:22:57 on dist-test-slave-wmd0"
server_key: "ba4a8c79a041562c68889e1ce5604e5e"
server_key_iv: "018e4380951dada930a2e0518cf302c1"
server_key_version: "encryptionkey@0"
I20260923 20:22:57.961254 25379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "83b558d2a1d541adb43b5927df9b34f4"
format_stamp: "Formatted at 2026-09-23 20:22:57 on dist-test-slave-wmd0"
server_key: "ba4a8c79a041562c68889e1ce5604e5e"
server_key_iv: "018e4380951dada930a2e0518cf302c1"
server_key_version: "encryptionkey@0"
I20260923 20:22:57.966602 25379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:22:57.970194 25394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:57.971728 25379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:22:57.971918 25379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "83b558d2a1d541adb43b5927df9b34f4"
format_stamp: "Formatted at 2026-09-23 20:22:57 on dist-test-slave-wmd0"
server_key: "ba4a8c79a041562c68889e1ce5604e5e"
server_key_iv: "018e4380951dada930a2e0518cf302c1"
server_key_version: "encryptionkey@0"
I20260923 20:22:57.972128 25379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:57.987149 25379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:57.991194 25379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:57.991443 25379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:57.992372 25379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:57.993764 25379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:57.993837 25379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:57.993918 25379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:57.993966 25379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:58.030078 25379 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:41469
I20260923 20:22:58.030122 25506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:41469 every 8 connection(s)
I20260923 20:22:58.031401 25379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:22:58.033654 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 25379
I20260923 20:22:58.033885 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:22:58.034317 14847 external_mini_cluster.cc:1428] Setting key 9060a6538a6b7c0642a2b436cf4a6474
Sep 23 20:22:58 dist-test-slave-wmd0 krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194977, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:58.053669 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.033959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:37603 (local address 127.14.127.254:39365)
0923 20:22:58.034172 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.034178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.035622 (+ 1444us) server_negotiation.cc:407] Connection header received
0923 20:22:58.037395 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.037401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.037486 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.037601 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:58.039143 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.039864 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.041452 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.042213 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.042326 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.046532 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:58.046541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:58.046549 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:58.046597 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:58.050258 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:58.051072 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:58.051081 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:58.051087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:58.051237 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:58.051655 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:58.051663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:58.051668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:58.052096 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:58.052254 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.052940 (+ 686us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.053212 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:22:58.054569 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.034521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:39365 (local address 127.14.127.194:37603)
0923 20:22:58.035458 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:58.035509 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:58.037121 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:58.037833 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:58.037842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:58.038340 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:58.038977 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.038994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.040014 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.040024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.041277 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.041288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.042468 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.042478 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.042850 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.043594 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:58.043626 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:58.046287 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:58.050426 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:58.050433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:58.050446 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:58.050934 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:58.051360 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:58.051363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:58.051365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:58.051537 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:58.052213 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:58.052219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:58.052564 (+ 345us) client_negotiation.cc:769] Sending connection context
0923 20:22:58.052781 (+ 217us) client_negotiation.cc:240] Negotiation successful
0923 20:22:58.053166 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":355,"threads_started":1}
I20260923 20:22:58.056247 25507 heartbeater.cc:347] Connected to a master server at 127.14.127.254:39365
I20260923 20:22:58.056586 25507 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:58.057293 25507 heartbeater.cc:511] Master 127.14.127.254:39365 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
I20260923 20:22:58.058874 25184 ts_manager.cc:194] Registered new tserver with Master: 83b558d2a1d541adb43b5927df9b34f4 (127.14.127.194:41469)
I20260923 20:22:58.059796 25184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:37603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:22:58.089653 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39365
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:22:58.408841 25512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:22:58.409121 25512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:22:58.409309 25512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:22:58.416741 25512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:22:58.416858 25512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:22:58.417038 25512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:22:58.426797 25512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:39365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:22:58.428068 25512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:22:58.429370 25512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:22:58.442440 25517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:58.442396 25518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:22:58.442487 25520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:22:58.443399 25512 server_base.cc:1034] running on GCE node
I20260923 20:22:58.443959 25512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:22:58.444839 25512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:22:58.446065 25512 hybrid_clock.cc:648] HybridClock initialized: now 1790194978446077 us; error 103 us; skew 500 ppm
I20260923 20:22:58.446376 25512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:22:58 dist-test-slave-wmd0 krb5kdc[25139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194978, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:22:58.449391 25512 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:22:58.450997 25512 webserver.cc:460] Webserver started at http://127.14.127.195:45035/ using document root <none> and password file <none>
I20260923 20:22:58.451670 25512 fs_manager.cc:359] Metadata directory not provided
I20260923 20:22:58.451848 25512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:22:58.452145 25512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:22:58.455605 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "a800d09228774dba901da7d632c778e8"
format_stamp: "Formatted at 2026-09-23 20:22:58 on dist-test-slave-wmd0"
server_key: "f88b0c1fdaaf320ca741f45aba8cf605"
server_key_iv: "781faa4893d50260f355ce03ed4140f7"
server_key_version: "encryptionkey@0"
I20260923 20:22:58.456385 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "a800d09228774dba901da7d632c778e8"
format_stamp: "Formatted at 2026-09-23 20:22:58 on dist-test-slave-wmd0"
server_key: "f88b0c1fdaaf320ca741f45aba8cf605"
server_key_iv: "781faa4893d50260f355ce03ed4140f7"
server_key_version: "encryptionkey@0"
I20260923 20:22:58.461655 25512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:22:58.465075 25527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:58.466431 25512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:22:58.466594 25512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "a800d09228774dba901da7d632c778e8"
format_stamp: "Formatted at 2026-09-23 20:22:58 on dist-test-slave-wmd0"
server_key: "f88b0c1fdaaf320ca741f45aba8cf605"
server_key_iv: "781faa4893d50260f355ce03ed4140f7"
server_key_version: "encryptionkey@0"
I20260923 20:22:58.466807 25512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:22:58.486671 25512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:22:58.490653 25512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:22:58.490955 25512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:22:58.491907 25512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:22:58.493287 25512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:22:58.493387 25512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:58.493537 25512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:22:58.493606 25512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:22:58.526754 25512 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:35367
I20260923 20:22:58.526851 25639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:35367 every 8 connection(s)
I20260923 20:22:58.528025 25512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:22:58.531437 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 25512
I20260923 20:22:58.531673 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:22:58.532220 14847 external_mini_cluster.cc:1428] Setting key d2a12635f08518268d6bde7090a6dc2f
Sep 23 20:22:58 dist-test-slave-wmd0 krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194978, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:22:58.548386 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.530655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:54419 (local address 127.14.127.254:39365)
0923 20:22:58.530864 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.530871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.532294 (+ 1423us) server_negotiation.cc:407] Connection header received
0923 20:22:58.534705 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.534711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.534790 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.534901 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:58.536627 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.537386 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.538948 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.539398 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.539503 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.542743 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:58.542748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:58.542751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:58.542786 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:58.545084 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:58.545813 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:58.545818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:58.545820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:58.545956 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:58.546461 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:58.546484 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:58.546487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:58.546832 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:58.546978 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.547819 (+ 841us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.548036 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:22:58.549316 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.531244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:39365 (local address 127.14.127.195:54419)
0923 20:22:58.532107 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:58.532166 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:58.534407 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:58.535110 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:58.535123 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:58.535770 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:22:58.536465 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.536483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.537514 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.537518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.538807 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.538829 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.539585 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.539590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.539784 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.540426 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:22:58.540453 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:22:58.542491 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:22:58.545222 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:58.545229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:58.545241 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:58.545656 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:58.546099 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:22:58.546104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:22:58.546108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:22:58.546344 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:22:58.546963 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:22:58.546971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:22:58.547389 (+ 418us) client_negotiation.cc:769] Sending connection context
0923 20:22:58.547733 (+ 344us) client_negotiation.cc:240] Negotiation successful
0923 20:22:58.548252 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":325,"threads_started":1}
I20260923 20:22:58.550575 25640 heartbeater.cc:347] Connected to a master server at 127.14.127.254:39365
I20260923 20:22:58.550935 25640 heartbeater.cc:464] Registering TS with master...
I20260923 20:22:58.551312 25373 heartbeater.cc:503] Master 127.14.127.254:39365 was elected leader, sending a full tablet report...
I20260923 20:22:58.551784 25640 heartbeater.cc:511] Master 127.14.127.254:39365 requested a full tablet report, sending...
I20260923 20:22:58.553428 25184 ts_manager.cc:194] Registered new tserver with Master: a800d09228774dba901da7d632c778e8 (127.14.127.195:35367)
I20260923 20:22:58.554265 25184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:54419
I20260923 20:22:58.560084 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:22:58.576990 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.564541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.14.127.254:39365)
0923 20:22:58.564717 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.564722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.565244 (+ 522us) server_negotiation.cc:407] Connection header received
0923 20:22:58.565377 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.565381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.565441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.565525 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:58.566996 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.567787 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.568984 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.569383 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.569478 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.571835 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:58.571840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:58.571843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:58.571877 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:58.574467 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:58.575144 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:58.575148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:58.575150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:58.575279 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:58.575628 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:58.575632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:58.575634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:58.576022 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:58.576205 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.576544 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.576707 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:22:58.580895 25184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:22:58.587718 25184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:22:58.622555 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.599598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.14.127.195:35367)
0923 20:22:58.613081 (+ 13483us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.613096 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.613136 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:22:58.613258 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.613269 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.613558 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.613748 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:58.614856 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.615900 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.619858 (+ 3958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.621235 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.621451 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.621625 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.621990 (+ 365us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.622159 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13237,"thread_start_us":14329,"threads_started":1}
I20260923 20:22:58.622650 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.599270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:35367 (local address 127.0.0.1:36022)
0923 20:22:58.612025 (+ 12755us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:58.612089 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:58.612318 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:58.614184 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:58.614191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:58.614216 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:58.614655 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.614670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.616032 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.616040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.619684 (+ 3644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.619699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.621462 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.621467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.621737 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.621806 (+ 69us) client_negotiation.cc:769] Sending connection context
0923 20:22:58.621941 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 20:22:58.622172 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12410,"mutex_wait_us":57,"thread_start_us":14543,"threads_started":1}
I20260923 20:22:58.625162 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.599405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:41469 (local address 127.0.0.1:38816)
0923 20:22:58.613612 (+ 14207us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:58.613630 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:58.613731 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:58.617239 (+ 3508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:58.617244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:58.617269 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:58.617711 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.617725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.619650 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.619660 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.622277 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.622293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.624331 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.624337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.624511 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.624542 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:22:58.624625 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 20:22:58.624758 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14097,"mutex_wait_us":50,"thread_start_us":12114,"threads_started":1}
I20260923 20:22:58.625936 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.599841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.14.127.194:41469)
0923 20:22:58.616469 (+ 16628us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.616477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.616503 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:22:58.616590 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.616598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.616823 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.617017 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:58.617914 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.619476 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.622478 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.623920 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.624021 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.624124 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.625443 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.625609 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16473,"thread_start_us":16322,"threads_started":1}
I20260923 20:22:58.626770 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.601503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33845 (local address 127.0.0.1:44314)
0923 20:22:58.618705 (+ 17202us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:58.618723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:58.618867 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:58.619384 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:58.619389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:58.619413 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:58.619865 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.619882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.621853 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.621857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.624058 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.624077 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.625862 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.625870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.626193 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.626225 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:22:58.626332 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:22:58.626447 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17034,"thread_start_us":16875,"threads_started":1}
I20260923 20:22:58.626900 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.601914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.14.127.193:33845)
0923 20:22:58.618154 (+ 16240us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.618166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.618836 (+ 670us) server_negotiation.cc:407] Connection header received
0923 20:22:58.619037 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.619043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.619206 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.619397 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:58.620094 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.621713 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.624227 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.625674 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.625875 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.626041 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.626388 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.626535 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16012,"thread_start_us":16672,"threads_started":1}
I20260923 20:22:58.626967 25575 tablet_service.cc:1467] Processing CreateTablet for tablet 591f41c591814e5b8d98e6f32e69dc1f (DEFAULT_TABLE table=test-table [id=3a4b1cd231d94a08a29d8e5f76b742f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:58.628494 25575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591f41c591814e5b8d98e6f32e69dc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:58.629127 25442 tablet_service.cc:1467] Processing CreateTablet for tablet 591f41c591814e5b8d98e6f32e69dc1f (DEFAULT_TABLE table=test-table [id=3a4b1cd231d94a08a29d8e5f76b742f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:58.630381 25442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591f41c591814e5b8d98e6f32e69dc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:58.630501 25308 tablet_service.cc:1467] Processing CreateTablet for tablet 591f41c591814e5b8d98e6f32e69dc1f (DEFAULT_TABLE table=test-table [id=3a4b1cd231d94a08a29d8e5f76b742f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:22:58.631834 25308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 591f41c591814e5b8d98e6f32e69dc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:22:58.645059 25659 tablet_bootstrap.cc:492] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Bootstrap starting.
I20260923 20:22:58.646561 25660 tablet_bootstrap.cc:492] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Bootstrap starting.
I20260923 20:22:58.649324 25660 tablet_bootstrap.cc:654] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:58.649560 25659 tablet_bootstrap.cc:654] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:58.650383 25660 log.cc:824] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:58.651069 25659 log.cc:824] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:58.652767 25658 tablet_bootstrap.cc:492] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Bootstrap starting.
I20260923 20:22:58.653627 25660 tablet_bootstrap.cc:492] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: No bootstrap required, opened a new log
I20260923 20:22:58.653903 25660 ts_tablet_manager.cc:1397] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 20:22:58.656606 25658 tablet_bootstrap.cc:654] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Neither blocks nor log segments found. Creating new log.
I20260923 20:22:58.657289 25659 tablet_bootstrap.cc:492] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: No bootstrap required, opened a new log
I20260923 20:22:58.657553 25659 ts_tablet_manager.cc:1397] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260923 20:22:58.657806 25658 log.cc:824] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:22:58.660758 25658 tablet_bootstrap.cc:492] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: No bootstrap required, opened a new log
I20260923 20:22:58.661005 25658 ts_tablet_manager.cc:1397] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260923 20:22:58.662869 25660 raft_consensus.cc:348] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.663324 25660 raft_consensus.cc:374] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:58.663425 25660 raft_consensus.cc:729] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02aa0694e3c34494ba06811f08b2ccc2, State: Initialized, Role: FOLLOWER
I20260923 20:22:58.664018 25660 consensus_queue.cc:260] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.665746 25660 ts_tablet_manager.cc:1428] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260923 20:22:58.668030 25659 raft_consensus.cc:348] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.668452 25659 raft_consensus.cc:374] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:58.668552 25659 raft_consensus.cc:729] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a800d09228774dba901da7d632c778e8, State: Initialized, Role: FOLLOWER
I20260923 20:22:58.669165 25659 consensus_queue.cc:260] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.670454 25640 heartbeater.cc:503] Master 127.14.127.254:39365 was elected leader, sending a full tablet report...
I20260923 20:22:58.671065 25659 ts_tablet_manager.cc:1428] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260923 20:22:58.671775 25658 raft_consensus.cc:348] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.672147 25658 raft_consensus.cc:374] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:22:58.672232 25658 raft_consensus.cc:729] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b558d2a1d541adb43b5927df9b34f4, State: Initialized, Role: FOLLOWER
I20260923 20:22:58.672748 25658 consensus_queue.cc:260] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.673780 25507 heartbeater.cc:503] Master 127.14.127.254:39365 was elected leader, sending a full tablet report...
I20260923 20:22:58.674294 25658 ts_tablet_manager.cc:1428] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260923 20:22:58.770905 25374 tablet.cc:2369] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:22:58.779621 25641 tablet.cc:2369] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:22:58.783354 25508 tablet.cc:2369] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:22:58.925165 25665 raft_consensus.cc:482] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:22:58.925534 25665 raft_consensus.cc:504] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.927452 25665 leader_election.cc:288] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b558d2a1d541adb43b5927df9b34f4 (127.14.127.194:41469), 02aa0694e3c34494ba06811f08b2ccc2 (127.14.127.193:33845)
I20260923 20:22:58.933863 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.928112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:35389 (local address 127.14.127.194:41469)
0923 20:22:58.928382 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.928389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.928411 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:22:58.928485 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.928490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.928567 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.928671 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:58.929090 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.930093 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.931975 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.933013 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.933125 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.933176 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.933553 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.933640 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:22:58.933933 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.927670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:41469 (local address 127.14.127.195:35389)
0923 20:22:58.927880 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:58.927906 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:58.928038 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:58.928696 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:58.928718 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:58.928748 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:58.928961 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.928972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.930220 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.930226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.931819 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.931831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.933153 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.933160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.933383 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.933418 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:22:58.933513 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 20:22:58.933649 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:22:58.934980 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.927837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33845 (local address 127.14.127.195:40973)
0923 20:22:58.928638 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0923 20:22:58.928661 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:22:58.928862 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:22:58.929235 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:22:58.929239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:22:58.929264 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:22:58.929617 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.929628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.930923 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.930927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.932537 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:22:58.932551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.934273 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.934281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:22:58.934451 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.934487 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:22:58.934561 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 20:22:58.934704 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"spinlock_wait_cycles":3456,"thread_start_us":460,"threads_started":1}
I20260923 20:22:58.935010 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.928457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:40973 (local address 127.14.127.193:33845)
0923 20:22:58.928646 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.928652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.928749 (+ 97us) server_negotiation.cc:407] Connection header received
0923 20:22:58.929007 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.929016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.929100 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.929182 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:22:58.929783 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.930766 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.932708 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.933735 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.933890 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.933972 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.934649 (+ 677us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.934782 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:22:58.934998 25462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591f41c591814e5b8d98e6f32e69dc1f" candidate_uuid: "a800d09228774dba901da7d632c778e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b558d2a1d541adb43b5927df9b34f4" is_pre_election: true
I20260923 20:22:58.935554 25462 raft_consensus.cc:2454] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a800d09228774dba901da7d632c778e8 in term 0.
I20260923 20:22:58.936098 25328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591f41c591814e5b8d98e6f32e69dc1f" candidate_uuid: "a800d09228774dba901da7d632c778e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02aa0694e3c34494ba06811f08b2ccc2" is_pre_election: true
I20260923 20:22:58.936313 25529 leader_election.cc:302] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b558d2a1d541adb43b5927df9b34f4, a800d09228774dba901da7d632c778e8; no voters:
I20260923 20:22:58.936656 25328 raft_consensus.cc:2454] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a800d09228774dba901da7d632c778e8 in term 0.
I20260923 20:22:58.936728 25665 raft_consensus.cc:2781] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:22:58.936890 25665 raft_consensus.cc:482] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:22:58.936990 25665 raft_consensus.cc:3037] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:58.939268 25665 raft_consensus.cc:504] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.939844 25665 leader_election.cc:288] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [CANDIDATE]: Term 1 election: Requested vote from peers 83b558d2a1d541adb43b5927df9b34f4 (127.14.127.194:41469), 02aa0694e3c34494ba06811f08b2ccc2 (127.14.127.193:33845)
I20260923 20:22:58.940269 25328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591f41c591814e5b8d98e6f32e69dc1f" candidate_uuid: "a800d09228774dba901da7d632c778e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02aa0694e3c34494ba06811f08b2ccc2"
I20260923 20:22:58.940284 25462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "591f41c591814e5b8d98e6f32e69dc1f" candidate_uuid: "a800d09228774dba901da7d632c778e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b558d2a1d541adb43b5927df9b34f4"
I20260923 20:22:58.940454 25328 raft_consensus.cc:3037] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:58.940469 25462 raft_consensus.cc:3037] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:22:58.942581 25462 raft_consensus.cc:2454] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a800d09228774dba901da7d632c778e8 in term 1.
I20260923 20:22:58.942587 25328 raft_consensus.cc:2454] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a800d09228774dba901da7d632c778e8 in term 1.
I20260923 20:22:58.943028 25529 leader_election.cc:302] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02aa0694e3c34494ba06811f08b2ccc2, 83b558d2a1d541adb43b5927df9b34f4, a800d09228774dba901da7d632c778e8; no voters:
I20260923 20:22:58.943367 25665 raft_consensus.cc:2781] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:22:58.943823 25665 raft_consensus.cc:686] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 1 LEADER]: Becoming Leader. State: Replica: a800d09228774dba901da7d632c778e8, State: Running, Role: LEADER
I20260923 20:22:58.944272 25665 consensus_queue.cc:237] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } }
I20260923 20:22:58.948979 25184 catalog_manager.cc:5495] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a800d09228774dba901da7d632c778e8 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "a800d09228774dba901da7d632c778e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a800d09228774dba901da7d632c778e8" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 } health_report { overall_health: UNKNOWN } } }
I20260923 20:22:58.988605 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:58.983219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.14.127.195:35367)
0923 20:22:58.983424 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:58.983431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:58.983613 (+ 182us) server_negotiation.cc:407] Connection header received
0923 20:22:58.983880 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:58.983884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:58.983967 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:58.984110 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:22:58.984499 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.985187 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.986493 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:58.986955 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:58.987084 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:58.987555 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:22:58.988050 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:22:58.988209 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:58.988267 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 20:22:58.988344 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:22:58.996137 25328 raft_consensus.cc:1261] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 1 FOLLOWER]: Refusing update from remote peer a800d09228774dba901da7d632c778e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:58.996163 25462 raft_consensus.cc:1261] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 1 FOLLOWER]: Refusing update from remote peer a800d09228774dba901da7d632c778e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:22:58.997042 25668 consensus_queue.cc:1035] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b558d2a1d541adb43b5927df9b34f4" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 41469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:58.997395 25665 consensus_queue.cc:1035] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02aa0694e3c34494ba06811f08b2ccc2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:22:59.011787 25672 mvcc.cc:205] Tried to move back new op lower bound from 7332638633959124992 to 7332638633759789056. Current Snapshot: MvccSnapshot[applied={T|T < 7332638633959124992}]
I20260923 20:22:59.021049 25184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:59.021298 25184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:22:59.024811 25184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bbd1d5ec18d4519b017220c557be516: Sending DeleteTablet for 3 replicas of tablet 591f41c591814e5b8d98e6f32e69dc1f
I20260923 20:22:59.025910 25442 tablet_service.cc:1514] Processing DeleteTablet for tablet 591f41c591814e5b8d98e6f32e69dc1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:59 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:38816
I20260923 20:22:59.025925 25575 tablet_service.cc:1514] Processing DeleteTablet for tablet 591f41c591814e5b8d98e6f32e69dc1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:59 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:36022
I20260923 20:22:59.026051 25308 tablet_service.cc:1514] Processing DeleteTablet for tablet 591f41c591814e5b8d98e6f32e69dc1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:22:59 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:44314
I20260923 20:22:59.028487 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 25245
I20260923 20:22:59.043309 25680 tablet_replica.cc:330] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: stopping tablet replica
I20260923 20:22:59.044023 25681 tablet_replica.cc:330] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: stopping tablet replica
I20260923 20:22:59.044711 25681 raft_consensus.cc:2229] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:22:59.045339 25680 raft_consensus.cc:2229] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:59.046271 25681 raft_consensus.cc:2258] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:59.046981 25680 raft_consensus.cc:2258] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:59.051314 25681 ts_tablet_manager.cc:1905] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:59.052152 25682 tablet_replica.cc:330] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: stopping tablet replica
I20260923 20:22:59.052282 25680 ts_tablet_manager.cc:1905] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:59.052570 25682 raft_consensus.cc:2229] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:22:59.053030 25682 raft_consensus.cc:2258] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:22:59.055393 25682 ts_tablet_manager.cc:1905] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:22:59.059338 25681 ts_tablet_manager.cc:1918] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:59.059511 25681 log.cc:1196] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/591f41c591814e5b8d98e6f32e69dc1f
I20260923 20:22:59.059993 25680 ts_tablet_manager.cc:1918] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:59.060072 25681 ts_tablet_manager.cc:1939] T 591f41c591814e5b8d98e6f32e69dc1f P a800d09228774dba901da7d632c778e8: Deleting consensus metadata
I20260923 20:22:59.060290 25680 log.cc:1196] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/591f41c591814e5b8d98e6f32e69dc1f
I20260923 20:22:59.060947 25680 ts_tablet_manager.cc:1939] T 591f41c591814e5b8d98e6f32e69dc1f P 83b558d2a1d541adb43b5927df9b34f4: Deleting consensus metadata
I20260923 20:22:59.062057 25171 catalog_manager.cc:4841] TS a800d09228774dba901da7d632c778e8 (127.14.127.195:35367): tablet 591f41c591814e5b8d98e6f32e69dc1f (table test-table [id=3a4b1cd231d94a08a29d8e5f76b742f8]) successfully deleted
I20260923 20:22:59.062233 25682 ts_tablet_manager.cc:1918] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:22:59.062409 25682 log.cc:1196] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/591f41c591814e5b8d98e6f32e69dc1f
I20260923 20:22:59.062942 25682 ts_tablet_manager.cc:1939] T 591f41c591814e5b8d98e6f32e69dc1f P 02aa0694e3c34494ba06811f08b2ccc2: Deleting consensus metadata
I20260923 20:22:59.063051 25169 catalog_manager.cc:4841] TS 83b558d2a1d541adb43b5927df9b34f4 (127.14.127.194:41469): tablet 591f41c591814e5b8d98e6f32e69dc1f (table test-table [id=3a4b1cd231d94a08a29d8e5f76b742f8]) successfully deleted
Sep 23 20:22:59 dist-test-slave-wmd0 krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:22:59.066037 25168 catalog_manager.cc:4841] TS 02aa0694e3c34494ba06811f08b2ccc2 (127.14.127.193:33845): tablet 591f41c591814e5b8d98e6f32e69dc1f (table test-table [id=3a4b1cd231d94a08a29d8e5f76b742f8]) successfully deleted
I20260923 20:22:59.072503 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:59.034782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.14.127.193:33845)
0923 20:22:59.035015 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:59.035024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:59.045366 (+ 10342us) server_negotiation.cc:407] Connection header received
0923 20:22:59.045796 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:59.045801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:59.045879 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:59.045947 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:59.047659 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:59.048888 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:59.050679 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:59.051365 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:59.051459 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:59.066712 (+ 15253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:59.066723 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:59.066736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:59.066775 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:59.069610 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:59.070350 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:59.070357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:59.070359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:59.070485 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:59.070961 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:59.070964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:59.070966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:59.071357 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:59.071661 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:59.071927 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:22:59.072149 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:22:59.073273 25368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:59.212846 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 25245
I20260923 20:22:59.230053 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 25379
Sep 23 20:22:59 dist-test-slave-wmd0 krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:22:59.246595 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:59.230897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.14.127.194:41469)
0923 20:22:59.231096 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:59.231102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:59.231126 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:22:59.231244 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:59.231249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:59.231346 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:59.231456 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:59.232664 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:59.233711 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:59.235102 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:59.235750 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:59.235867 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:59.239845 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:59.239868 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:59.239891 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:59.239943 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:59.243762 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:59.244441 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:59.244452 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:59.244457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:59.244617 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:59.244953 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:59.244958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:59.244962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:59.245427 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:59.245726 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:59.245934 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 20:22:59.246161 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:22:59.247346 25502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:59.337996 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 25379
I20260923 20:22:59.355422 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 25512
Sep 23 20:22:59 dist-test-slave-wmd0 krb5kdc[25139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:22:59.373291 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:22:59.356219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.14.127.195:35367)
0923 20:22:59.356426 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:22:59.356432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:22:59.356454 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:22:59.356572 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:22:59.356578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:22:59.356682 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:22:59.356814 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:22:59.358081 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:59.359350 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:59.360676 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:22:59.361399 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:22:59.361527 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:22:59.365816 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:22:59.365831 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:22:59.365853 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:22:59.365914 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:22:59.370036 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:59.371003 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:59.371013 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:59.371016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:59.371222 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:22:59.371574 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:22:59.371581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:22:59.371586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:22:59.372089 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:22:59.372411 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0923 20:22:59.372596 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 20:22:59.372822 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 20:22:59.373999 25635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:59.497639 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 25512
I20260923 20:22:59.515549 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 25152
I20260923 20:22:59.516671 25214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:22:59.641920 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 25152
2026-09-23T20:22:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4503 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:22:59 dist-test-slave-wmd0 krb5kdc[25690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:22:59 dist-test-slave-wmd0 krb5kdc[25690](info): set up 2 sockets
Sep 23 20:22:59 dist-test-slave-wmd0 krb5kdc[25690](info): commencing operation
krb5kdc: starting...
W20260923 20:23:00.743782 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:23:00 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:23:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.127.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.127.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/rangeradmin_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/rangeradmin_127.14.127.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.127.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.127.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/rangerlookup_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/rangerlookup_127.14.127.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.127.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.127.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/HTTP_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/HTTP_127.14.127.212@KRBTEST.COM.keytab.
I20260923 20:23:01.073527 14847 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFITuhD/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/postgres -l logfile start
2026-09-23 20:23:03.714 UTC [25720] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 20:23:03.714 UTC [25720] LOG: listening on IPv4 address "127.14.127.212", port 52557
2026-09-23 20:23:03.718 UTC [25720] LOG: listening on Unix socket "/tmp/.s.PGSQL.52557"
2026-09-23 20:23:03.738 UTC [25723] LOG: database system was shut down at 2026-09-23 20:23:02 UTC
2026-09-23 20:23:03.742 UTC [25720] LOG: database system is ready to accept connections
I20260923 20:23:04.723351 14847 mini_postgres.cc:96] Postgres bound to 52557
2026-09-23 20:23:04.742 UTC [25731] FATAL: database "slave" does not exist
127.14.127.212:52557 - accepting connections
I20260923 20:23:04.743646 14847 mini_ranger.cc:161] Starting Ranger...
I20260923 20:23:04.771031 14847 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 20:23:05.286007 14847 mini_ranger.cc:87] Created ranger Postgres database
I20260923 20:23:05.286152 14847 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-admin
2026-09-23 20:23:06,188 [I] DB FLAVOR :POSTGRES
2026-09-23 20:23:06,189 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:23:06,189 [I] Checking connection..
2026-09-23 20:23:06,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 20:23:06,590 [I] Checking connection passed.
2026-09-23 20:23:06,591 [I] --------- Verifying version history table ---------
2026-09-23 20:23:06,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:23:06,844 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:23:06,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:23:07,082 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:23:07,082 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 20:23:07,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 20:23:07.331 UTC [25795] WARNING: there is no transaction in progress
2026-09-23 20:23:07,338 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 20:23:07,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:23:07,573 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 20:23:07,573 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:23:07,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 20:23:07,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 20:23:08,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wmd0.c.gcp-upstream.internal','N') ;"
2026-09-23 20:23:08,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:23:08,579 [I] Table x_portal_user does not exist in database ranger
2026-09-23 20:23:08,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:23:08,833 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 20:23:08,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:23:09,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:23:09,397 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 20:23:09,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 20:23:10.401 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.418 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.435 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.452 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.469 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.487 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.542 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.548 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.563 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.573 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.584 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.598 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.607 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.613 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.620 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:10.627 UTC [25903] WARNING: there is no transaction in progress
2026-09-23 20:23:11,270 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 20:23:11,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 20:23:11,520 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 20:23:11,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:23:11,783 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 20:23:11,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:23:12,035 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 20:23:12,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:23:12,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:23:12,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wmd0.c.gcp-upstream.internal';"
2026-09-23 20:23:12,788 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 20:23:12,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 20:23:13,035 [I] DB_PATCHES have already been applied
I20260923 20:23:13.041687 14847 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFITuhD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 20:23:13.041776 14847 mini_ranger.cc:193] Using host: 127.14.127.212
I20260923 20:23:13.056061 14847 mini_ranger.cc:227] Ranger admin URL: http://127.14.127.212:35537
Sep 23, 2026 8:23:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:23:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:23:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 8:23:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 8:23:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 8:23:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 8:23:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 20:23:15.236898 14847 mini_ranger.cc:160] Time spent starting Ranger: real 10.493s user 0.001s sys 0.046s
Sep 23, 2026 8:23:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 8:23:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 8:23:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 8:23:37 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 8:23:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 20:23:38.591696 14847 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.127.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.127.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/rangerkms_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/rangerkms_127.14.127.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.127.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.127.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.127.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/HTTP_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.127.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/HTTP_127.14.127.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 20:23:38 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195018, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 20:23:38.760951 14847 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 20:23:38.782585 14847 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 20:23:39.635135 14847 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 20:23:39.635295 14847 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms
I20260923 20:23:39.635412 14847 mini_ranger_kms.cc:227] Using postgres at 127.14.127.212:52557
2026-09-23 20:23:40,489 [I] DB FLAVOR :POSTGRES
2026-09-23 20:23:40,489 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:23:40,489 [I] Checking connection
2026-09-23 20:23:40,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 20:23:40,726 [I] connection success
2026-09-23 20:23:40,726 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 20:23:40,726 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 20:23:40,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 20:23:40,963 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 20:23:40,963 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:23:40,963 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 20:23:40,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:52557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 20:23:41,233 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 20:23:41.702170 14847 mini_ranger_kms.cc:310] Created kms service
I20260923 20:23:41.901731 14847 mini_ranger_kms.cc:326] Created kudu user
I20260923 20:23:41.944775 14847 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 20:23:41 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195018, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:23:42.368882 14847 mini_ranger_kms.cc:384] Added ranger policy
I20260923 20:23:42.369999 14847 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 20:23:42.370085 14847 mini_ranger_kms.cc:240] Using host: 127.14.127.212
I20260923 20:23:42.391191 14847 mini_ranger_kms.cc:276] Ranger KMS PID: 26085
I20260923 20:23:42.391386 14847 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.127.212:59021
Sep 23, 2026 8:23:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:23:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:23:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 8:23:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 8:23:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 8:23:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 20:23:44 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790195024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.127.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:23:44.707516 14847 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.947s user 0.147s sys 0.380s
I20260923 20:23:44.707780 14847 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 20:23:44.707983 14847 mini_ranger_kms.cc:401] 127.14.127.212:59021/kms/v1/keys
Sep 23 20:23:45 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.127.212@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
Sep 23 20:23:45 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.127.212@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
Sep 23 20:23:47 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:23:47.858371 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:42697
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:42697
--ranger_config_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:59021/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:23:48.199491 26135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:23:48.199793 26135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:23:48.199946 26135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:23:48.207541 26135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:23:48.207656 26135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:23:48.207733 26135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:23:48.207842 26135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:23:48.207909 26135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:23:48.218288 26135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.127.212:59021/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:42697
--ranger_config_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:23:48.219669 26135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:23:48.221014 26135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:23:48.233705 26141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:23:48.233851 26140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:23:48.233840 26143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:23:48.234488 26135 server_base.cc:1034] running on GCE node
I20260923 20:23:48.235425 26135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:23:48.236886 26135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:23:48.238104 26135 hybrid_clock.cc:648] HybridClock initialized: now 1790195028238062 us; error 39 us; skew 500 ppm
I20260923 20:23:48.238472 26135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:23:48 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195028, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:23:48.248039 26135 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:23:48.249689 26135 webserver.cc:460] Webserver started at http://127.14.127.254:38347/ using document root <none> and password file <none>
I20260923 20:23:48.250381 26135 fs_manager.cc:359] Metadata directory not provided
I20260923 20:23:48.250514 26135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:23:48.250874 26135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:23:48 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195028, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:23:48.302232 26135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "0b8635ae5b6740558fcbc9f0607c97b6"
format_stamp: "Formatted at 2026-09-23 20:23:48 on dist-test-slave-wmd0"
server_key: "22501b6b4e78b952def085c3ec97d8ad"
server_key_iv: "05abb872ab31219c51ba71065e5933db"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:48.303125 26135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "0b8635ae5b6740558fcbc9f0607c97b6"
format_stamp: "Formatted at 2026-09-23 20:23:48 on dist-test-slave-wmd0"
server_key: "22501b6b4e78b952def085c3ec97d8ad"
server_key_iv: "05abb872ab31219c51ba71065e5933db"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:48.308727 26135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:23:48.325616 26150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:48.327323 26135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:23:48.327534 26135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "0b8635ae5b6740558fcbc9f0607c97b6"
format_stamp: "Formatted at 2026-09-23 20:23:48 on dist-test-slave-wmd0"
server_key: "22501b6b4e78b952def085c3ec97d8ad"
server_key_iv: "05abb872ab31219c51ba71065e5933db"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:48.327772 26135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:23:48.359371 26135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:23:48.363682 26135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:23:48.364010 26135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:23:48.385113 26135 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:42697
I20260923 20:23:48.385136 26201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:42697 every 8 connection(s)
I20260923 20:23:48.386530 26135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:23:48.391342 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:23:48.395891 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 26135
I20260923 20:23:48.396126 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:23:48.396761 14847 external_mini_cluster.cc:1428] Setting key 087a314164529378f4daafe9c6bdf287
I20260923 20:23:48.405464 26202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6: Bootstrap starting.
I20260923 20:23:48.409468 26202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6: Neither blocks nor log segments found. Creating new log.
I20260923 20:23:48.410970 26202 log.cc:824] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:23:48.416962 26202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6: No bootstrap required, opened a new log
Sep 23 20:23:48 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:23:48.428057 26202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8635ae5b6740558fcbc9f0607c97b6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42697 } }
I20260923 20:23:48.428545 26202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:23:48.428651 26202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8635ae5b6740558fcbc9f0607c97b6, State: Initialized, Role: FOLLOWER
I20260923 20:23:48.429378 26202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8635ae5b6740558fcbc9f0607c97b6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42697 } }
I20260923 20:23:48.429644 26202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:23:48.429833 26202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:23:48.430019 26202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:23:48.433429 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:48.399040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.14.127.254:42697)
0923 20:23:48.413520 (+ 14480us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:48.413539 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:48.413591 (+ 52us) server_negotiation.cc:407] Connection header received
0923 20:23:48.415778 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:48.415801 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:48.416191 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:48.416609 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:23:48.418001 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:48.419378 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:48.420633 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:48.421329 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:48.421541 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:48.425280 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:23:48.425289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:23:48.425308 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:23:48.425344 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:23:48.428821 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:48.429730 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:48.429738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:48.429749 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:48.429914 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:48.430373 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:48.430379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:48.430382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:48.431097 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:23:48.431453 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:48.431893 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:23:48.432452 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14215,"thread_start_us":14352,"threads_started":1}
I20260923 20:23:48.433517 26202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8635ae5b6740558fcbc9f0607c97b6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42697 } }
I20260923 20:23:48.434118 26202 leader_election.cc:302] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b8635ae5b6740558fcbc9f0607c97b6; no voters:
I20260923 20:23:48.434792 26202 leader_election.cc:288] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:23:48.434922 26207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:23:48.438966 26207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0b8635ae5b6740558fcbc9f0607c97b6, State: Running, Role: LEADER
I20260923 20:23:48.439515 26207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8635ae5b6740558fcbc9f0607c97b6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42697 } }
I20260923 20:23:48.443472 26202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:23:48.445148 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b8635ae5b6740558fcbc9f0607c97b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8635ae5b6740558fcbc9f0607c97b6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42697 } } }
I20260923 20:23:48.445639 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [sys.catalog]: This master's current role is: LEADER
I20260923 20:23:48.446183 26208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b8635ae5b6740558fcbc9f0607c97b6. Latest consensus state: current_term: 1 leader_uuid: "0b8635ae5b6740558fcbc9f0607c97b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8635ae5b6740558fcbc9f0607c97b6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42697 } } }
I20260923 20:23:48.446512 26208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6 [sys.catalog]: This master's current role is: LEADER
I20260923 20:23:48.455940 26212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:23:48.461761 26212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:23:48.487427 26212 catalog_manager.cc:1347] Generated new cluster ID: 91dff4dc45c84044bfc95fd38b7f5b8f
I20260923 20:23:48.487555 26212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:23:48.510448 26202 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:23:48.511023 26212 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:23:48.512940 26212 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:23:48.541378 26212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6: Generated new TSK 0
I20260923 20:23:48.542810 26212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:23:49.451007 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:59021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42697
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:23:49 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790195029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 20:23:49.892949 26247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:23:49.893455 26247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:23:49.893666 26247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:23:49.905536 26247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:23:49.905697 26247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:23:49.905907 26247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:23:49.921875 26247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.127.212:59021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:23:49.923729 26247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:23:49.925527 26247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:23:49.946455 26257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:23:49.947403 26258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:23:49.954442 26261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:23:49.955603 26247 server_base.cc:1034] running on GCE node
I20260923 20:23:49.956367 26247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:23:49.957615 26247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:23:49.958940 26247 hybrid_clock.cc:648] HybridClock initialized: now 1790195029958871 us; error 57 us; skew 500 ppm
I20260923 20:23:49.959321 26247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:23:49 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:23:49.963560 26247 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:23:49.966491 26247 webserver.cc:460] Webserver started at http://127.14.127.193:46635/ using document root <none> and password file <none>
I20260923 20:23:49.967550 26247 fs_manager.cc:359] Metadata directory not provided
I20260923 20:23:49.967687 26247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:23:49.968050 26247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:23:49 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:23:49.991882 26247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "264892a645ea4fd584c36f2654e950bc"
format_stamp: "Formatted at 2026-09-23 20:23:49 on dist-test-slave-wmd0"
server_key: "9e68617aea6cd561c96b226673b0877c"
server_key_iv: "5428686dfa99f2b8b064dcffc1318e28"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:49.993427 26247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "264892a645ea4fd584c36f2654e950bc"
format_stamp: "Formatted at 2026-09-23 20:23:49 on dist-test-slave-wmd0"
server_key: "9e68617aea6cd561c96b226673b0877c"
server_key_iv: "5428686dfa99f2b8b064dcffc1318e28"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:50.004102 26247 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260923 20:23:50.055438 26267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:50.057596 26247 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.010s sys 0.005s
I20260923 20:23:50.057803 26247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "264892a645ea4fd584c36f2654e950bc"
format_stamp: "Formatted at 2026-09-23 20:23:49 on dist-test-slave-wmd0"
server_key: "9e68617aea6cd561c96b226673b0877c"
server_key_iv: "5428686dfa99f2b8b064dcffc1318e28"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:50.058054 26247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:23:50.080930 26247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:23:50.089721 26247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:23:50.090081 26247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:23:50.091219 26247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:23:50.093161 26247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:23:50.093268 26247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:50.093417 26247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:23:50.093502 26247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:50.158994 26247 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:43549
I20260923 20:23:50.159013 26379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:43549 every 8 connection(s)
I20260923 20:23:50.160553 26247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:23:50.167570 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 26247
I20260923 20:23:50.167840 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:23:50.168823 14847 external_mini_cluster.cc:1428] Setting key b4424b50c046ff4be341084c599aad56
Sep 23 20:23:50 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:23:50.211112 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:50.166211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42697 (local address 127.14.127.193:35183)
0923 20:23:50.180411 (+ 14200us) negotiation.cc:107] Waiting for socket to connect
0923 20:23:50.180474 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:23:50.193129 (+ 12655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:23:50.193988 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:23:50.193998 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:23:50.194450 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:23:50.195135 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:50.195166 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:50.196689 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:50.196695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:50.198115 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:50.198136 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:50.199063 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:50.199071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:50.199309 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:50.200015 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:23:50.200038 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:23:50.202061 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:23:50.206134 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:23:50.206143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:23:50.206177 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:23:50.206803 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:23:50.207325 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:23:50.207330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:23:50.207335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:23:50.207597 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:23:50.208287 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:23:50.208296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:23:50.208752 (+ 456us) client_negotiation.cc:769] Sending connection context
0923 20:23:50.209052 (+ 300us) client_negotiation.cc:240] Negotiation successful
0923 20:23:50.209632 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13779,"thread_start_us":23511,"threads_started":1}
I20260923 20:23:50.213184 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:50.165365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:35183 (local address 127.14.127.254:42697)
0923 20:23:50.180637 (+ 15272us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:50.180643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:50.180676 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:23:50.193475 (+ 12799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:50.193482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:50.193594 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:50.193723 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:23:50.195332 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:50.196472 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:50.198281 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:50.198825 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:50.198965 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:50.202297 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:23:50.202306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:23:50.202316 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:23:50.202357 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:23:50.205958 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:50.206982 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:50.206987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:50.206991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:50.207175 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:50.207750 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:50.207755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:50.207759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:50.208114 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:23:50.212355 (+ 4241us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:50.212542 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 20:23:50.212786 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15155,"thread_start_us":15308,"threads_started":1}
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:23:50.214942 26380 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42697
I20260923 20:23:50.215288 26380 heartbeater.cc:464] Registering TS with master...
I20260923 20:23:50.216193 26380 heartbeater.cc:511] Master 127.14.127.254:42697 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
I20260923 20:23:50.218750 26167 ts_manager.cc:194] Registered new tserver with Master: 264892a645ea4fd584c36f2654e950bc (127.14.127.193:43549)
I20260923 20:23:50.220829 26167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:35183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:23:50.249914 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:59021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42697
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:23:50 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
Sep 23 20:23:50 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195029, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
W20260923 20:23:50.728600 26386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:23:50.729087 26386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:23:50.729279 26386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:23:50.740468 26386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:23:50.740674 26386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:23:50.740875 26386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:23:50.756439 26386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.127.212:59021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:23:50.758332 26386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:23:50.760130 26386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:23:50.803177 26393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:23:50.805068 26394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:23:50.811105 26397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:23:50.811637 26386 server_base.cc:1034] running on GCE node
I20260923 20:23:50.812655 26386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:23:50.817500 26386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:23:50.818771 26386 hybrid_clock.cc:648] HybridClock initialized: now 1790195030818669 us; error 103 us; skew 500 ppm
I20260923 20:23:50.819192 26386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:23:50 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195030, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:23:50.823256 26386 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:23:50.840739 26386 webserver.cc:460] Webserver started at http://127.14.127.194:39637/ using document root <none> and password file <none>
I20260923 20:23:50.841825 26386 fs_manager.cc:359] Metadata directory not provided
I20260923 20:23:50.842039 26386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:23:50.842445 26386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:23:50 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195030, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:23:50.873629 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "3e8528ec27634d4cbb63f5fc941345b7"
format_stamp: "Formatted at 2026-09-23 20:23:50 on dist-test-slave-wmd0"
server_key: "a79be97351c58177cd77aa04c9df90e0"
server_key_iv: "f289f4d68bd2295a38fa3b77241297e2"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:50.883277 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "3e8528ec27634d4cbb63f5fc941345b7"
format_stamp: "Formatted at 2026-09-23 20:23:50 on dist-test-slave-wmd0"
server_key: "a79be97351c58177cd77aa04c9df90e0"
server_key_iv: "f289f4d68bd2295a38fa3b77241297e2"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:50.904680 26386 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.004s sys 0.012s
I20260923 20:23:50.961488 26405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:50.963562 26386 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.005s sys 0.000s
I20260923 20:23:50.963835 26386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "3e8528ec27634d4cbb63f5fc941345b7"
format_stamp: "Formatted at 2026-09-23 20:23:50 on dist-test-slave-wmd0"
server_key: "a79be97351c58177cd77aa04c9df90e0"
server_key_iv: "f289f4d68bd2295a38fa3b77241297e2"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:50.964154 26386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:23:51.015334 26386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:23:51.020462 26386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:23:51.020843 26386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:23:51.035033 26386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:23:51.043190 26386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:23:51.043397 26386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:23:51.043596 26386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:23:51.043696 26386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:51.207623 26386 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:43091
I20260923 20:23:51.207625 26522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:43091 every 8 connection(s)
I20260923 20:23:51.209116 26386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:23:51.219226 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 26386
I20260923 20:23:51.219434 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:23:51.219945 14847 external_mini_cluster.cc:1428] Setting key 8db1c3597befab5de75d802ee3f5baca
I20260923 20:23:51.224781 26380 heartbeater.cc:503] Master 127.14.127.254:42697 was elected leader, sending a full tablet report...
Sep 23 20:23:51 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195030, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:23:51.242846 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.213673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42697 (local address 127.14.127.194:47183)
0923 20:23:51.217890 (+ 4217us) negotiation.cc:107] Waiting for socket to connect
0923 20:23:51.217948 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 20:23:51.220481 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:23:51.221577 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:23:51.221590 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:23:51.222228 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:23:51.223122 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.223153 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.224761 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.224768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.226796 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.226824 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.228164 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.228173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.228515 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.229358 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:23:51.229392 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:23:51.232520 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:23:51.236772 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:23:51.236785 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:23:51.236806 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:23:51.237453 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:23:51.238669 (+ 1216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:23:51.238677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:23:51.238684 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:23:51.238940 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:23:51.239786 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:23:51.239798 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:23:51.240279 (+ 481us) client_negotiation.cc:769] Sending connection context
0923 20:23:51.240613 (+ 334us) client_negotiation.cc:240] Negotiation successful
0923 20:23:51.241196 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3904,"thread_start_us":3644,"threads_started":1}
I20260923 20:23:51.244637 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.214370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:47183 (local address 127.14.127.254:42697)
0923 20:23:51.217283 (+ 2913us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:51.217293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:51.218465 (+ 1172us) server_negotiation.cc:407] Connection header received
0923 20:23:51.220965 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:51.220971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:51.221051 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:51.221155 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:23:51.223325 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.224546 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.226969 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.227695 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.227821 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.232787 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:23:51.232797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:23:51.232817 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:23:51.232870 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:23:51.236597 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:51.238325 (+ 1728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:51.238334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:51.238340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:51.238500 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:51.239086 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:51.239094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:51.239100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:51.239566 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:23:51.239712 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:51.243793 (+ 4081us) server_negotiation.cc:299] Negotiation successful
0923 20:23:51.244129 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2742,"thread_start_us":2642,"threads_started":1}
I20260923 20:23:51.246140 26523 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42697
I20260923 20:23:51.246507 26523 heartbeater.cc:464] Registering TS with master...
I20260923 20:23:51.247287 26523 heartbeater.cc:511] Master 127.14.127.254:42697 requested a full tablet report, sending...
I20260923 20:23:51.248806 26167 ts_manager.cc:194] Registered new tserver with Master: 3e8528ec27634d4cbb63f5fc941345b7 (127.14.127.194:43091)
I20260923 20:23:51.250080 26167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:47183
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:23:51.287089 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:59021/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42697
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:23:51.638209 26529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:23:51.638502 26529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:23:51.638728 26529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:23:51.646106 26529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:23:51.646221 26529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:23:51.646394 26529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:23:51.656184 26529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.127.212:59021/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:23:51.657531 26529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:23:51.658934 26529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:23:51.672552 26534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:23:51.672622 26535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:23:51.673231 26537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:23:51.673543 26529 server_base.cc:1034] running on GCE node
I20260923 20:23:51.674261 26529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:23:51.675444 26529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:23:51.676671 26529 hybrid_clock.cc:648] HybridClock initialized: now 1790195031676615 us; error 47 us; skew 500 ppm
I20260923 20:23:51.677034 26529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:23:51 dist-test-slave-wmd0 krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195031, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:23:51.680855 26529 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:23:51.682497 26529 webserver.cc:460] Webserver started at http://127.14.127.195:41895/ using document root <none> and password file <none>
I20260923 20:23:51.683377 26529 fs_manager.cc:359] Metadata directory not provided
I20260923 20:23:51.683525 26529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:23:51.683930 26529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:23:51 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195031, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:23:51.702570 26529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "cc258667f1a14c25a2233cf4a31193e5"
format_stamp: "Formatted at 2026-09-23 20:23:51 on dist-test-slave-wmd0"
server_key: "d269a6c6d2903fd0e34cac9fc33e82de"
server_key_iv: "5c3ec3ce6b1312155308779368026eaf"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:51.703658 26529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "cc258667f1a14c25a2233cf4a31193e5"
format_stamp: "Formatted at 2026-09-23 20:23:51 on dist-test-slave-wmd0"
server_key: "d269a6c6d2903fd0e34cac9fc33e82de"
server_key_iv: "5c3ec3ce6b1312155308779368026eaf"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:51.710922 26529 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260923 20:23:51.728571 26544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:51.730374 26529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 20:23:51.730561 26529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "cc258667f1a14c25a2233cf4a31193e5"
format_stamp: "Formatted at 2026-09-23 20:23:51 on dist-test-slave-wmd0"
server_key: "d269a6c6d2903fd0e34cac9fc33e82de"
server_key_iv: "5c3ec3ce6b1312155308779368026eaf"
server_key_version: "kuduclusterkey@0"
I20260923 20:23:51.730872 26529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:23:51.752816 26529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:23:51.756826 26529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:23:51.757071 26529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:23:51.757988 26529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:23:51.759366 26529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:23:51.759438 26529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:51.759523 26529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:23:51.759567 26529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:23:51.792780 26529 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:46869
I20260923 20:23:51.792817 26656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:46869 every 8 connection(s)
I20260923 20:23:51.794085 26529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:23:51.794581 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 26529
I20260923 20:23:51.794862 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:23:51.795327 14847 external_mini_cluster.cc:1428] Setting key f8438cecf8ba15fac96686b5e914a8f4
Sep 23 20:23:51 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195031, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:23:51.814669 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.797454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:51439 (local address 127.14.127.254:42697)
0923 20:23:51.798721 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:51.798726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:51.799462 (+ 736us) server_negotiation.cc:407] Connection header received
0923 20:23:51.801207 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:51.801211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:51.801272 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:51.801364 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:23:51.802848 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.803584 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.805066 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.805521 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.805619 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.808784 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:23:51.808789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:23:51.808793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:23:51.808824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:23:51.811976 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:51.812658 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:51.812662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:51.812664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:51.812788 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:51.813160 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:51.813163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:51.813165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:51.813433 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:23:51.813539 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:51.814161 (+ 622us) server_negotiation.cc:299] Negotiation successful
0923 20:23:51.814351 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":937,"threads_started":1}
I20260923 20:23:51.815578 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.797957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42697 (local address 127.14.127.195:51439)
0923 20:23:51.799305 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0923 20:23:51.799351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:23:51.800950 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:23:51.801585 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:23:51.801593 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:23:51.802016 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:23:51.802604 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.802695 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.803705 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.803709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.804930 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.804949 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.805708 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.805713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.805907 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.806512 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:23:51.806533 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:23:51.808586 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:23:51.812101 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:23:51.812108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:23:51.812121 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:23:51.812549 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:23:51.812877 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:23:51.812880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:23:51.812882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:23:51.813065 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:23:51.813541 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:23:51.813547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:23:51.813846 (+ 299us) client_negotiation.cc:769] Sending connection context
0923 20:23:51.814063 (+ 217us) client_negotiation.cc:240] Negotiation successful
0923 20:23:51.814438 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":822,"threads_started":1}
I20260923 20:23:51.816941 26657 heartbeater.cc:347] Connected to a master server at 127.14.127.254:42697
I20260923 20:23:51.817198 26657 heartbeater.cc:464] Registering TS with master...
I20260923 20:23:51.817826 26657 heartbeater.cc:511] Master 127.14.127.254:42697 requested a full tablet report, sending...
I20260923 20:23:51.819197 26167 ts_manager.cc:194] Registered new tserver with Master: cc258667f1a14c25a2233cf4a31193e5 (127.14.127.195:46869)
I20260923 20:23:51.820007 26167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:51439
I20260923 20:23:51.823475 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:23:51.839880 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.827885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.14.127.254:42697)
0923 20:23:51.828112 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:51.828118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:51.828461 (+ 343us) server_negotiation.cc:407] Connection header received
0923 20:23:51.828586 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:51.828589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:51.828659 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:51.828738 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:23:51.830084 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.830885 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.832254 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.832696 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.832786 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.835006 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:23:51.835011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:23:51.835014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:23:51.835043 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:23:51.837595 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:51.838204 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:51.838207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:51.838209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:51.838319 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:51.838653 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:51.838663 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:51.838666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:51.838909 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:23:51.839072 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:51.839427 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 20:23:51.839639 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:23:51.843957 26167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:23:51.851603 26167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:23:51.875677 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.864913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:43091 (local address 127.0.0.1:45454)
0923 20:23:51.865962 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0923 20:23:51.865984 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:23:51.866117 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:23:51.868391 (+ 2274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:23:51.868396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:23:51.868418 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:23:51.868725 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.868733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.870901 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.870906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.872987 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.873014 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.874559 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.874564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.874841 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.874878 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:23:51.875025 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 20:23:51.875229 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":25,"thread_start_us":1708,"threads_started":1}
I20260923 20:23:51.875716 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.865692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.14.127.194:43091)
0923 20:23:51.867512 (+ 1820us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:51.867521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:51.867551 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:23:51.867625 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:51.867630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:51.867931 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:51.868131 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:23:51.868883 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.870735 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.873151 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.874348 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.874523 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.874736 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:51.875208 (+ 472us) server_negotiation.cc:299] Negotiation successful
0923 20:23:51.875389 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1677,"thread_start_us":1579,"threads_started":1}
I20260923 20:23:51.879205 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.866917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43549 (local address 127.0.0.1:34316)
0923 20:23:51.869280 (+ 2363us) negotiation.cc:107] Waiting for socket to connect
0923 20:23:51.869302 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:23:51.869422 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:23:51.870901 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:23:51.870906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:23:51.870936 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:23:51.871474 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.871485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.873907 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.873914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.875997 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.876010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.878443 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.878447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.878552 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.878573 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:23:51.878701 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 20:23:51.878843 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2225,"mutex_wait_us":31,"thread_start_us":2098,"threads_started":1}
I20260923 20:23:51.879249 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.867507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.14.127.193:43549)
0923 20:23:51.868131 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:51.868139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:51.869876 (+ 1737us) server_negotiation.cc:407] Connection header received
0923 20:23:51.870326 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:51.870341 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:51.870551 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:51.870768 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:23:51.871643 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.873730 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.876863 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.878271 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.878458 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.878604 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:51.878764 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 20:23:51.878921 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":287,"threads_started":1}
I20260923 20:23:51.879724 26458 tablet_service.cc:1467] Processing CreateTablet for tablet 00e79416e54043e0a9806860704d105c (DEFAULT_TABLE table=test-table [id=8204acfc8f85446f909a4bee302232da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:23:51.880389 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.864172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:46869 (local address 127.0.0.1:35180)
0923 20:23:51.864976 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:23:51.865021 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:23:51.865223 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:23:51.870550 (+ 5327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:23:51.870556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:23:51.870582 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:23:51.871059 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.871070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.872878 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.872886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.875319 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:51.875333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.879581 (+ 4248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.879587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:51.879746 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.879781 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:23:51.879887 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:23:51.880007 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":35,"thread_start_us":373,"threads_started":1}
I20260923 20:23:51.880424 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:51.864480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.14.127.195:46869)
0923 20:23:51.869938 (+ 5458us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:51.869946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:51.869975 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:23:51.870058 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:51.870065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:51.870367 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:51.870539 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:23:51.871263 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.872762 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.876172 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:51.879377 (+ 3205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:51.879585 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:51.879738 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:51.879925 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 20:23:51.880073 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5302,"thread_start_us":5147,"threads_started":1}
I20260923 20:23:51.881016 26458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e79416e54043e0a9806860704d105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:23:51.884013 26592 tablet_service.cc:1467] Processing CreateTablet for tablet 00e79416e54043e0a9806860704d105c (DEFAULT_TABLE table=test-table [id=8204acfc8f85446f909a4bee302232da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:23:51.885159 26592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e79416e54043e0a9806860704d105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:23:51.886137 26315 tablet_service.cc:1467] Processing CreateTablet for tablet 00e79416e54043e0a9806860704d105c (DEFAULT_TABLE table=test-table [id=8204acfc8f85446f909a4bee302232da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:23:51.887531 26315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e79416e54043e0a9806860704d105c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:23:51.897192 26676 tablet_bootstrap.cc:492] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Bootstrap starting.
I20260923 20:23:51.900101 26676 tablet_bootstrap.cc:654] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Neither blocks nor log segments found. Creating new log.
I20260923 20:23:51.900940 26677 tablet_bootstrap.cc:492] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Bootstrap starting.
I20260923 20:23:51.901413 26676 log.cc:824] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:23:51.903702 26679 tablet_bootstrap.cc:492] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Bootstrap starting.
I20260923 20:23:51.904539 26677 tablet_bootstrap.cc:654] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Neither blocks nor log segments found. Creating new log.
I20260923 20:23:51.905069 26676 tablet_bootstrap.cc:492] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: No bootstrap required, opened a new log
I20260923 20:23:51.905340 26676 ts_tablet_manager.cc:1397] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 20:23:51.905735 26677 log.cc:824] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:23:51.907302 26679 tablet_bootstrap.cc:654] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Neither blocks nor log segments found. Creating new log.
I20260923 20:23:51.908202 26679 log.cc:824] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Log is configured to *not* fsync() on all Append() calls
I20260923 20:23:51.908797 26677 tablet_bootstrap.cc:492] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: No bootstrap required, opened a new log
I20260923 20:23:51.909055 26677 ts_tablet_manager.cc:1397] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260923 20:23:51.911007 26679 tablet_bootstrap.cc:492] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: No bootstrap required, opened a new log
I20260923 20:23:51.911262 26679 ts_tablet_manager.cc:1397] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 20:23:51.915573 26676 raft_consensus.cc:348] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:51.916006 26676 raft_consensus.cc:374] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:23:51.916096 26676 raft_consensus.cc:729] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc258667f1a14c25a2233cf4a31193e5, State: Initialized, Role: FOLLOWER
I20260923 20:23:51.916723 26676 consensus_queue.cc:260] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:51.917856 26657 heartbeater.cc:503] Master 127.14.127.254:42697 was elected leader, sending a full tablet report...
I20260923 20:23:51.918497 26676 ts_tablet_manager.cc:1428] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260923 20:23:51.919355 26677 raft_consensus.cc:348] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:51.919409 26679 raft_consensus.cc:348] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:51.919750 26677 raft_consensus.cc:374] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:23:51.919831 26679 raft_consensus.cc:374] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:23:51.919864 26677 raft_consensus.cc:729] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8528ec27634d4cbb63f5fc941345b7, State: Initialized, Role: FOLLOWER
I20260923 20:23:51.919935 26679 raft_consensus.cc:729] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 264892a645ea4fd584c36f2654e950bc, State: Initialized, Role: FOLLOWER
I20260923 20:23:51.920454 26677 consensus_queue.cc:260] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:51.920523 26679 consensus_queue.cc:260] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:51.921700 26523 heartbeater.cc:503] Master 127.14.127.254:42697 was elected leader, sending a full tablet report...
I20260923 20:23:51.922317 26677 ts_tablet_manager.cc:1428] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260923 20:23:51.924293 26679 ts_tablet_manager.cc:1428] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260923 20:23:51.933974 26381 tablet.cc:2369] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:23:51.967628 26524 tablet.cc:2369] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:23:52.012084 26682 raft_consensus.cc:482] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:23:52.012378 26682 raft_consensus.cc:504] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:52.014038 26682 leader_election.cc:288] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8528ec27634d4cbb63f5fc941345b7 (127.14.127.194:43091), 264892a645ea4fd584c36f2654e950bc (127.14.127.193:43549)
I20260923 20:23:52.022070 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:52.014433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43549 (local address 127.14.127.195:36385)
0923 20:23:52.014732 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 20:23:52.014754 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:23:52.014979 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:23:52.015618 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:23:52.015625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:23:52.015656 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:23:52.015864 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:52.015877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.017462 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.017467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:52.019765 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:52.019778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.021309 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.021315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:52.021508 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:52.021541 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:23:52.021635 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 20:23:52.021765 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":222}
I20260923 20:23:52.022233 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:52.014637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:36385 (local address 127.14.127.193:43549)
0923 20:23:52.014872 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:52.014878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:52.014903 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:23:52.015223 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:52.015231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:52.015376 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:52.015594 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:23:52.016078 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.017318 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.019951 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.021099 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.021313 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:52.021386 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:52.021703 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 20:23:52.021935 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:23:52.022805 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:52.014483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:50933 (local address 127.14.127.194:43091)
0923 20:23:52.014711 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:52.014717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:52.015701 (+ 984us) server_negotiation.cc:407] Connection header received
0923 20:23:52.015959 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:52.015966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:52.016050 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:52.016156 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:23:52.016817 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.018067 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.020495 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.021739 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.021889 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:52.021955 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:52.022457 (+ 502us) server_negotiation.cc:299] Negotiation successful
0923 20:23:52.022579 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:23:52.023247 26335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e79416e54043e0a9806860704d105c" candidate_uuid: "cc258667f1a14c25a2233cf4a31193e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "264892a645ea4fd584c36f2654e950bc" is_pre_election: true
I20260923 20:23:52.023751 26335 raft_consensus.cc:2454] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc258667f1a14c25a2233cf4a31193e5 in term 0.
I20260923 20:23:52.024426 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:52.014917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:43091 (local address 127.14.127.195:50933)
0923 20:23:52.015606 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 20:23:52.015622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:23:52.015799 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:23:52.016213 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:23:52.016220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:23:52.016270 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:23:52.016630 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:52.016641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.018267 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.018275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:52.020328 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:23:52.020340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.022101 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.022108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:23:52.022296 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:52.022331 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:23:52.022429 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 20:23:52.022564 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":4,"thread_start_us":344,"threads_started":1}
I20260923 20:23:52.024523 26547 leader_election.cc:302] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 264892a645ea4fd584c36f2654e950bc, cc258667f1a14c25a2233cf4a31193e5; no voters:
I20260923 20:23:52.024986 26682 raft_consensus.cc:2781] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:23:52.025257 26682 raft_consensus.cc:482] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:23:52.025381 26682 raft_consensus.cc:3037] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:23:52.025434 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e79416e54043e0a9806860704d105c" candidate_uuid: "cc258667f1a14c25a2233cf4a31193e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8528ec27634d4cbb63f5fc941345b7" is_pre_election: true
I20260923 20:23:52.025977 26478 raft_consensus.cc:2454] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc258667f1a14c25a2233cf4a31193e5 in term 0.
I20260923 20:23:52.027961 26682 raft_consensus.cc:504] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:52.028523 26682 leader_election.cc:288] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [CANDIDATE]: Term 1 election: Requested vote from peers 3e8528ec27634d4cbb63f5fc941345b7 (127.14.127.194:43091), 264892a645ea4fd584c36f2654e950bc (127.14.127.193:43549)
I20260923 20:23:52.028928 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e79416e54043e0a9806860704d105c" candidate_uuid: "cc258667f1a14c25a2233cf4a31193e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8528ec27634d4cbb63f5fc941345b7"
I20260923 20:23:52.028990 26335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e79416e54043e0a9806860704d105c" candidate_uuid: "cc258667f1a14c25a2233cf4a31193e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "264892a645ea4fd584c36f2654e950bc"
I20260923 20:23:52.029119 26478 raft_consensus.cc:3037] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:23:52.029210 26335 raft_consensus.cc:3037] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:23:52.031550 26335 raft_consensus.cc:2454] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc258667f1a14c25a2233cf4a31193e5 in term 1.
I20260923 20:23:52.031592 26478 raft_consensus.cc:2454] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc258667f1a14c25a2233cf4a31193e5 in term 1.
I20260923 20:23:52.031984 26547 leader_election.cc:302] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 264892a645ea4fd584c36f2654e950bc, cc258667f1a14c25a2233cf4a31193e5; no voters:
I20260923 20:23:52.032292 26682 raft_consensus.cc:2781] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:23:52.032734 26682 raft_consensus.cc:686] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 1 LEADER]: Becoming Leader. State: Replica: cc258667f1a14c25a2233cf4a31193e5, State: Running, Role: LEADER
I20260923 20:23:52.033262 26682 consensus_queue.cc:237] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } }
I20260923 20:23:52.038141 26165 catalog_manager.cc:5495] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc258667f1a14c25a2233cf4a31193e5 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "cc258667f1a14c25a2233cf4a31193e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc258667f1a14c25a2233cf4a31193e5" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 46869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 } health_report { overall_health: UNKNOWN } } }
W20260923 20:23:52.046846 26658 tablet.cc:2369] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:23:52.069159 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:52.063852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.14.127.195:46869)
0923 20:23:52.064056 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:52.064060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:52.064079 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:23:52.064268 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:52.064272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:52.064351 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:52.064471 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:23:52.064805 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.065517 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.067024 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.067443 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.067581 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:52.068102 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:23:52.068593 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:23:52.068767 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:52.068818 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 20:23:52.068895 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:23:52.077826 26335 raft_consensus.cc:1261] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 1 FOLLOWER]: Refusing update from remote peer cc258667f1a14c25a2233cf4a31193e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:23:52.077829 26478 raft_consensus.cc:1261] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 1 FOLLOWER]: Refusing update from remote peer cc258667f1a14c25a2233cf4a31193e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:23:52.078879 26686 consensus_queue.cc:1035] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8528ec27634d4cbb63f5fc941345b7" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:23:52.079221 26682 consensus_queue.cc:1035] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "264892a645ea4fd584c36f2654e950bc" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:23:52.095257 26689 mvcc.cc:205] Tried to move back new op lower bound from 7332638851380146176 to 7332638851212333056. Current Snapshot: MvccSnapshot[applied={T|T < 7332638851380146176}]
I20260923 20:23:52.107810 26165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:23:52.108075 26165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:23:52.111703 26165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b8635ae5b6740558fcbc9f0607c97b6: Sending DeleteTablet for 3 replicas of tablet 00e79416e54043e0a9806860704d105c
I20260923 20:23:52.112908 26592 tablet_service.cc:1514] Processing DeleteTablet for tablet 00e79416e54043e0a9806860704d105c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:23:52 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:35180
I20260923 20:23:52.112901 26458 tablet_service.cc:1514] Processing DeleteTablet for tablet 00e79416e54043e0a9806860704d105c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:23:52 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:45454
I20260923 20:23:52.113183 26315 tablet_service.cc:1514] Processing DeleteTablet for tablet 00e79416e54043e0a9806860704d105c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:23:52 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:34316
I20260923 20:23:52.115845 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 26247
I20260923 20:23:52.125586 26699 tablet_replica.cc:330] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: stopping tablet replica
I20260923 20:23:52.126160 26699 raft_consensus.cc:2229] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:23:52.126773 26699 raft_consensus.cc:2258] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:23:52.127666 26700 tablet_replica.cc:330] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: stopping tablet replica
I20260923 20:23:52.128166 26700 raft_consensus.cc:2229] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:23:52.129086 26700 raft_consensus.cc:2258] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:23:52.131979 26700 ts_tablet_manager.cc:1905] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:23:52.132144 26699 ts_tablet_manager.cc:1905] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:23:52.135571 26701 tablet_replica.cc:330] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: stopping tablet replica
I20260923 20:23:52.136680 26701 raft_consensus.cc:2229] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:23:52.137262 26701 raft_consensus.cc:2258] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:23:52.139431 26700 ts_tablet_manager.cc:1918] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:23:52.139595 26700 log.cc:1196] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/00e79416e54043e0a9806860704d105c
I20260923 20:23:52.139619 26701 ts_tablet_manager.cc:1905] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:23:52.140164 26700 ts_tablet_manager.cc:1939] T 00e79416e54043e0a9806860704d105c P cc258667f1a14c25a2233cf4a31193e5: Deleting consensus metadata
I20260923 20:23:52.140161 26699 ts_tablet_manager.cc:1918] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:23:52.140316 26699 log.cc:1196] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/00e79416e54043e0a9806860704d105c
I20260923 20:23:52.140832 26699 ts_tablet_manager.cc:1939] T 00e79416e54043e0a9806860704d105c P 3e8528ec27634d4cbb63f5fc941345b7: Deleting consensus metadata
I20260923 20:23:52.142112 26151 catalog_manager.cc:4841] TS cc258667f1a14c25a2233cf4a31193e5 (127.14.127.195:46869): tablet 00e79416e54043e0a9806860704d105c (table test-table [id=8204acfc8f85446f909a4bee302232da]) successfully deleted
I20260923 20:23:52.142496 26152 catalog_manager.cc:4841] TS 3e8528ec27634d4cbb63f5fc941345b7 (127.14.127.194:43091): tablet 00e79416e54043e0a9806860704d105c (table test-table [id=8204acfc8f85446f909a4bee302232da]) successfully deleted
I20260923 20:23:52.147625 26701 ts_tablet_manager.cc:1918] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:23:52.147884 26701 log.cc:1196] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/00e79416e54043e0a9806860704d105c
I20260923 20:23:52.148424 26701 ts_tablet_manager.cc:1939] T 00e79416e54043e0a9806860704d105c P 264892a645ea4fd584c36f2654e950bc: Deleting consensus metadata
I20260923 20:23:52.150357 26153 catalog_manager.cc:4841] TS 264892a645ea4fd584c36f2654e950bc (127.14.127.193:43549): tablet 00e79416e54043e0a9806860704d105c (table test-table [id=8204acfc8f85446f909a4bee302232da]) successfully deleted
Sep 23 20:23:52 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:23:52.158336 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:52.117658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.14.127.193:43549)
0923 20:23:52.117885 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:52.117894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:52.130408 (+ 12514us) server_negotiation.cc:407] Connection header received
0923 20:23:52.142799 (+ 12391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:52.142805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:52.142905 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:52.142995 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:23:52.144857 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.145603 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.147147 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.147573 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.147716 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:52.152242 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:23:52.152254 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:23:52.152268 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:23:52.152302 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:23:52.155287 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:52.156046 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:52.156054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:52.156056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:52.156201 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:52.156629 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:52.156633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:52.156635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:52.156997 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:23:52.157331 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:52.157653 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 20:23:52.157902 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:23:52.159168 26375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:23:52.266299 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 26247
I20260923 20:23:52.284765 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 26386
Sep 23 20:23:52 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:23:52.302357 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:52.285657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.14.127.194:43091)
0923 20:23:52.285838 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:52.285842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:52.285861 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:23:52.286023 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:52.286027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:52.286088 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:52.286171 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:23:52.287897 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.288675 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.290679 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.291163 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.291265 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:52.296395 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:23:52.296417 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:23:52.296433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:23:52.296472 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:23:52.299532 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:52.300230 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:52.300244 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:52.300246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:52.300376 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:52.300774 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:52.300779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:52.300784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:52.301130 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:23:52.301437 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:52.301794 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 20:23:52.302004 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:23:52.303095 26518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:23:52.398958 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 26386
I20260923 20:23:52.422559 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 26529
Sep 23 20:23:52 dist-test-slave-wmd0 krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:23:52.441417 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:23:52.423432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.14.127.195:46869)
0923 20:23:52.423629 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:23:52.423635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:23:52.423661 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:23:52.423816 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:23:52.423822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:23:52.423906 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:23:52.424051 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:23:52.425348 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.426513 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.428195 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:23:52.428881 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:23:52.429026 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:23:52.433602 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:23:52.433619 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:23:52.433641 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:23:52.433702 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:23:52.438281 (+ 4579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:52.439115 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:52.439126 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:52.439135 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:52.439300 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:23:52.439691 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:23:52.439696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:23:52.439699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:23:52.440172 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:23:52.440476 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 20:23:52.440760 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 20:23:52.441011 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:23:52.442221 26652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:23:52.576227 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 26529
I20260923 20:23:52.606154 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 26135
I20260923 20:23:52.607667 26197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:23:52.718564 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 26135
I20260923 20:23:52.739899 14847 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 20:23:53.076119 14847 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 20:23:53.076361 14847 mini_ranger.cc:66] Stopping Ranger...
I20260923 20:23:53.422582 14847 mini_ranger.cc:68] Stopped Ranger
2026-09-23 20:23:53.423 UTC [25720] LOG: received smart shutdown request
2026-09-23 20:23:53.426 UTC [25720] LOG: background worker "logical replication launcher" (PID 25729) exited with exit code 1
2026-09-23 20:23:53.426 UTC [25724] LOG: shutting down
2026-09-23 20:23:53.599 UTC [25720] LOG: database system is shut down
2026-09-23T20:23:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54082 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:23:53 dist-test-slave-wmd0 krb5kdc[26715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:23:53 dist-test-slave-wmd0 krb5kdc[26715](info): set up 2 sockets
Sep 23 20:23:53 dist-test-slave-wmd0 krb5kdc[26715](info): commencing operation
krb5kdc: starting...
W20260923 20:23:54.839098 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:23:54 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:23:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.127.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.127.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/rangeradmin_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/rangeradmin_127.14.127.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.127.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.127.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/rangerlookup_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/rangerlookup_127.14.127.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.127.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.127.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/HTTP_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/HTTP_127.14.127.212@KRBTEST.COM.keytab.
I20260923 20:23:55.156801 14847 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFITuhD/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/postgres -l logfile start
2026-09-23 20:23:57.663 UTC [26745] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 20:23:57.663 UTC [26745] LOG: listening on IPv4 address "127.14.127.212", port 46469
2026-09-23 20:23:57.667 UTC [26745] LOG: listening on Unix socket "/tmp/.s.PGSQL.46469"
2026-09-23 20:23:57.689 UTC [26748] LOG: database system was shut down at 2026-09-23 20:23:56 UTC
2026-09-23 20:23:57.693 UTC [26745] LOG: database system is ready to accept connections
I20260923 20:23:58.666740 14847 mini_postgres.cc:96] Postgres bound to 46469
2026-09-23 20:23:58.688 UTC [26756] FATAL: database "slave" does not exist
127.14.127.212:46469 - accepting connections
I20260923 20:23:58.689285 14847 mini_ranger.cc:161] Starting Ranger...
I20260923 20:23:58.715900 14847 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 20:23:59.211314 14847 mini_ranger.cc:87] Created ranger Postgres database
I20260923 20:23:59.211436 14847 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-admin
2026-09-23 20:23:59,479 [I] DB FLAVOR :POSTGRES
2026-09-23 20:23:59,479 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:23:59,480 [I] Checking connection..
2026-09-23 20:23:59,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 20:23:59,738 [I] Checking connection passed.
2026-09-23 20:23:59,738 [I] --------- Verifying version history table ---------
2026-09-23 20:23:59,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:24:00,010 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:24:00,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:24:00,261 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:24:00,261 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 20:24:00,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 20:24:00.489 UTC [26820] WARNING: there is no transaction in progress
2026-09-23 20:24:00,496 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 20:24:00,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:24:00,765 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 20:24:00,765 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:24:00,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 20:24:01,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 20:24:01,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wmd0.c.gcp-upstream.internal','N') ;"
2026-09-23 20:24:01,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:24:01,766 [I] Table x_portal_user does not exist in database ranger
2026-09-23 20:24:01,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:24:02,021 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 20:24:02,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:24:02,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:24:02,522 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 20:24:02,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 20:24:03.472 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.489 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.506 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.522 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.538 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.557 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.613 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.620 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.633 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.643 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.653 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.667 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.677 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.683 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.690 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:03.696 UTC [26928] WARNING: there is no transaction in progress
2026-09-23 20:24:04,324 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 20:24:04,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 20:24:04,557 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 20:24:04,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:24:04,793 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 20:24:04,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:24:05,045 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 20:24:05,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:24:05,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:24:05,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wmd0.c.gcp-upstream.internal';"
2026-09-23 20:24:05,759 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 20:24:05,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 20:24:06,033 [I] DB_PATCHES have already been applied
I20260923 20:24:06.040048 14847 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFITuhD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 20:24:06.040171 14847 mini_ranger.cc:193] Using host: 127.14.127.212
I20260923 20:24:06.056427 14847 mini_ranger.cc:227] Ranger admin URL: http://127.14.127.212:49921
Sep 23, 2026 8:24:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:24:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:24:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 8:24:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 8:24:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 8:24:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 8:24:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 20:24:07.204694 14847 mini_ranger.cc:160] Time spent starting Ranger: real 8.515s user 0.002s sys 0.039s
Sep 23, 2026 8:24:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 8:24:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 23, 2026 8:24:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 8:24:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 8:24:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 20:24:29.480587 14847 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.127.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.127.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/rangerkms_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.127.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/rangerkms_127.14.127.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.127.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.127.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.127.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/HTTP_127.14.127.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.127.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/HTTP_127.14.127.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 20:24:29 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195069, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 20:24:29.661144 14847 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 20:24:29.684597 14847 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 20:24:30.665220 14847 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 20:24:30.665323 14847 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms
I20260923 20:24:30.665352 14847 mini_ranger_kms.cc:227] Using postgres at 127.14.127.212:46469
2026-09-23 20:24:31,025 [I] DB FLAVOR :POSTGRES
2026-09-23 20:24:31,026 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:24:31,026 [I] Checking connection
2026-09-23 20:24:31,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 20:24:31,269 [I] connection success
2026-09-23 20:24:31,269 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 20:24:31,269 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 20:24:31,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 20:24:31,513 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 20:24:31,513 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:24:31,514 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 20:24:31,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.127.212:46469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 20:24:31,784 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 20:24:32.299924 14847 mini_ranger_kms.cc:310] Created kms service
I20260923 20:24:32.481010 14847 mini_ranger_kms.cc:326] Created kudu user
I20260923 20:24:32.530022 14847 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 20:24:32 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195069, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:32.930130 14847 mini_ranger_kms.cc:384] Added ranger policy
I20260923 20:24:32.930485 14847 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFITuhD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFITuhD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 20:24:32.930534 14847 mini_ranger_kms.cc:240] Using host: 127.14.127.212
I20260923 20:24:32.945152 14847 mini_ranger_kms.cc:276] Ranger KMS PID: 27108
I20260923 20:24:32.945288 14847 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.127.212:38939
Sep 23, 2026 8:24:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:24:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:24:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 8:24:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 8:24:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 8:24:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 20:24:35.254963 14847 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.594s user 0.082s sys 0.294s
I20260923 20:24:35.255220 14847 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 20:24:35.255385 14847 mini_ranger_kms.cc:401] 127.14.127.212:38939/kms/v1/keys
Sep 23 20:24:35 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790195075, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.127.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 20:24:35 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195075, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.127.212@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
Sep 23 20:24:36 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195075, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.127.212@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
Sep 23 20:24:38 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:24:38.672116 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37217
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:37217
--ranger_config_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:38939/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:39.009037 27159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:39.009392 27159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:39.009567 27159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:39.016808 27159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:24:39.016968 27159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:39.017077 27159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:39.017192 27159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:24:39.017287 27159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:24:39.027132 27159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.127.212:38939/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:37217
--ranger_config_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:39.028446 27159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:39.029779 27159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:39.041988 27165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:39.042347 27164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:39.042342 27167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:39.043138 27159 server_base.cc:1034] running on GCE node
I20260923 20:24:39.043843 27159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:39.045200 27159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:39.046492 27159 hybrid_clock.cc:648] HybridClock initialized: now 1790195079046454 us; error 34 us; skew 500 ppm
I20260923 20:24:39.046886 27159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:39 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195079, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:39.054478 27159 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:24:39.056063 27159 webserver.cc:460] Webserver started at http://127.14.127.254:34421/ using document root <none> and password file <none>
I20260923 20:24:39.056694 27159 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:39.056790 27159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:39.057037 27159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:24:39 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195079, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:39.106959 27159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "d358b507c890404e959d730b3a3f7768"
format_stamp: "Formatted at 2026-09-23 20:24:39 on dist-test-slave-wmd0"
server_key: "1c25cfa0491cc84f77f031bd8fd47216"
server_key_iv: "aef1ef43e14c40b47999593512b6f4ab"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:39.108014 27159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "d358b507c890404e959d730b3a3f7768"
format_stamp: "Formatted at 2026-09-23 20:24:39 on dist-test-slave-wmd0"
server_key: "1c25cfa0491cc84f77f031bd8fd47216"
server_key_iv: "aef1ef43e14c40b47999593512b6f4ab"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:39.113693 27159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:24:39.134136 27174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:39.135778 27159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:24:39.135977 27159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "d358b507c890404e959d730b3a3f7768"
format_stamp: "Formatted at 2026-09-23 20:24:39 on dist-test-slave-wmd0"
server_key: "1c25cfa0491cc84f77f031bd8fd47216"
server_key_iv: "aef1ef43e14c40b47999593512b6f4ab"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:39.136220 27159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:39.154876 27159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:39.159296 27159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:39.159621 27159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:39.179713 27159 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:37217
I20260923 20:24:39.179762 27225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:37217 every 8 connection(s)
I20260923 20:24:39.181064 27159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:24:39.185123 27226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:39.188410 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 27159
I20260923 20:24:39.188617 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:24:39.189404 14847 external_mini_cluster.cc:1428] Setting key 360fe58a6336e2655dda1b97a5fe583c
I20260923 20:24:39.200239 27226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Bootstrap starting.
I20260923 20:24:39.205479 27226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:39.207058 27226 log.cc:824] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:39.210845 27226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: No bootstrap required, opened a new log
I20260923 20:24:39.219815 27226 raft_consensus.cc:348] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } }
I20260923 20:24:39.220258 27226 raft_consensus.cc:374] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:39.220393 27226 raft_consensus.cc:729] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d358b507c890404e959d730b3a3f7768, State: Initialized, Role: FOLLOWER
Sep 23 20:24:39 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:39.221182 27226 consensus_queue.cc:260] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } }
I20260923 20:24:39.221490 27226 raft_consensus.cc:388] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:24:39.221669 27226 raft_consensus.cc:482] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:24:39.221921 27226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:39.224699 27226 raft_consensus.cc:504] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } }
I20260923 20:24:39.225248 27226 leader_election.cc:302] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d358b507c890404e959d730b3a3f7768; no voters:
I20260923 20:24:39.225998 27226 leader_election.cc:288] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:24:39.226115 27231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:24:39.230257 27231 raft_consensus.cc:686] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 1 LEADER]: Becoming Leader. State: Replica: d358b507c890404e959d730b3a3f7768, State: Running, Role: LEADER
I20260923 20:24:39.231127 27231 consensus_queue.cc:237] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } }
I20260923 20:24:39.232334 27229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:39.191569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.14.127.254:37217)
0923 20:24:39.206534 (+ 14965us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:39.206551 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:39.206601 (+ 50us) server_negotiation.cc:407] Connection header received
0923 20:24:39.209074 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:39.209104 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:39.209576 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:39.210042 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:39.211668 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:39.213704 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:39.215579 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:39.216720 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:39.216925 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:39.221919 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:39.221990 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:39.222011 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:39.222057 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:39.225956 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:39.226881 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:39.226889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:39.226899 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:39.227103 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:39.227564 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:39.227569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:39.227572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:39.228262 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:39.229679 (+ 1417us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:39.229982 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 20:24:39.230415 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14733,"thread_start_us":14519,"threads_started":1}
I20260923 20:24:39.238046 27226 sys_catalog.cc:564] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:24:39.240042 27232 sys_catalog.cc:455] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d358b507c890404e959d730b3a3f7768. Latest consensus state: current_term: 1 leader_uuid: "d358b507c890404e959d730b3a3f7768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } } }
I20260923 20:24:39.240147 27233 sys_catalog.cc:455] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d358b507c890404e959d730b3a3f7768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } } }
I20260923 20:24:39.240442 27232 sys_catalog.cc:458] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:39.240521 27233 sys_catalog.cc:458] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:39.247221 27239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:24:39.254855 27239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:24:39.273190 27239 catalog_manager.cc:1347] Generated new cluster ID: 219278164f274a84817b97b2b2125a7e
I20260923 20:24:39.273331 27239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:24:39.290349 27239 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:24:39.300572 27239 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:24:39.311820 27226 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:24:39.334990 27239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Generated new TSK 0
I20260923 20:24:39.336625 27239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:24:40.143316 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:38939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:37217
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:24:40 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790195080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 20:24:40.587809 27272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:40.588142 27272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:40.588323 27272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:40.596284 27272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:40.596427 27272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:40.596609 27272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:24:40.609663 27272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.127.212:38939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:40.611337 27272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:40.613029 27272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:40.638353 27282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:40.638150 27283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:40.640888 27285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:40.641376 27272 server_base.cc:1034] running on GCE node
I20260923 20:24:40.642066 27272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:40.643170 27272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:40.644413 27272 hybrid_clock.cc:648] HybridClock initialized: now 1790195080644351 us; error 58 us; skew 500 ppm
I20260923 20:24:40.644750 27272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:40 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:40.648139 27272 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:24:40.649820 27272 webserver.cc:460] Webserver started at http://127.14.127.193:36367/ using document root <none> and password file <none>
I20260923 20:24:40.650538 27272 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:40.650717 27272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:40.651036 27272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:24:40 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:40.674963 27272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "36c3b8bec27945ef88973234de559a3e"
format_stamp: "Formatted at 2026-09-23 20:24:40 on dist-test-slave-wmd0"
server_key: "c9fcdfe9f0b100359cf9d9f759ce5328"
server_key_iv: "4490a88f50a0b104f26f1b1d446d94e5"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:40.676183 27272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "36c3b8bec27945ef88973234de559a3e"
format_stamp: "Formatted at 2026-09-23 20:24:40 on dist-test-slave-wmd0"
server_key: "c9fcdfe9f0b100359cf9d9f759ce5328"
server_key_iv: "4490a88f50a0b104f26f1b1d446d94e5"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:40.704489 27272 fs_manager.cc:691] Time spent creating directory manager: real 0.028s user 0.004s sys 0.012s
I20260923 20:24:40.740056 27292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:40.742203 27272 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.011s sys 0.003s
I20260923 20:24:40.742501 27272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "36c3b8bec27945ef88973234de559a3e"
format_stamp: "Formatted at 2026-09-23 20:24:40 on dist-test-slave-wmd0"
server_key: "c9fcdfe9f0b100359cf9d9f759ce5328"
server_key_iv: "4490a88f50a0b104f26f1b1d446d94e5"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:40.742895 27272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:40.798062 27272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:40.803431 27272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:40.803830 27272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:40.811137 27272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:40.819167 27272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:40.819335 27272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:24:40.819528 27272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:40.819635 27272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:24:40 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:41.382468 27404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:40079 every 8 connection(s)
I20260923 20:24:41.382881 27272 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:40079
I20260923 20:24:41.384704 27272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:24:41.387904 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 27272
I20260923 20:24:41.388167 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:24:41.389279 14847 external_mini_cluster.cc:1428] Setting key e3d6f5c3da9b2a1fb6d3f3dd73e47902
Sep 23 20:24:41 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:41.444793 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:41.393756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:37217 (local address 127.14.127.193:59375)
0923 20:24:41.406316 (+ 12560us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:41.406377 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:41.409168 (+ 2791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:41.424383 (+ 15215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:41.424398 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:41.425131 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:41.426110 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:41.426143 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:41.427747 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:41.427752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:41.429572 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:41.429587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:41.430819 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:41.430826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:41.431095 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:41.431933 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:41.431962 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:41.434719 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:41.439361 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:41.439373 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:41.439392 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:41.440036 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:41.440948 (+ 912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:41.440957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:41.440961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:41.441242 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:41.442025 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:41.442037 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:41.442511 (+ 474us) client_negotiation.cc:769] Sending connection context
0923 20:24:41.442862 (+ 351us) client_negotiation.cc:240] Negotiation successful
0923 20:24:41.443382 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12203,"thread_start_us":15710,"threads_started":1}
I20260923 20:24:41.449249 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:41.393065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:59375 (local address 127.14.127.254:37217)
0923 20:24:41.423517 (+ 30452us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:41.423523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:41.423546 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:24:41.423629 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:41.423635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:41.423723 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:41.423843 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:41.426347 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:41.427542 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:41.429773 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:41.430368 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:41.430480 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:41.434993 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:41.435003 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:41.435011 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:41.435069 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:41.439171 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:41.440225 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:41.440232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:41.440237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:41.440398 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:41.441392 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:41.441397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:41.441401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:41.441787 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:41.441921 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:41.448597 (+ 6676us) server_negotiation.cc:299] Negotiation successful
0923 20:24:41.448850 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30306,"thread_start_us":30163,"threads_started":1}
I20260923 20:24:41.451728 27405 heartbeater.cc:347] Connected to a master server at 127.14.127.254:37217
I20260923 20:24:41.452119 27405 heartbeater.cc:464] Registering TS with master...
Sep 23 20:24:41 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:41.453106 27405 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:24:41.455785 27191 ts_manager.cc:194] Registered new tserver with Master: 36c3b8bec27945ef88973234de559a3e (127.14.127.193:40079)
I20260923 20:24:41.457860 27191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:59375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
W20260923 20:24:41.493825 14847 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.14.127.194: real 0.101s user 0.002s sys 0.016s
I20260923 20:24:41.494656 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:38939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:37217
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:41.878396 27412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:41.878737 27412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:41.878914 27412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:41.886085 27412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:41.886233 27412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:41.886413 27412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:24:41.896047 27412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.127.212:38939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:41.897316 27412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:41.898559 27412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:41.911341 27426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:41.911399 27425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:41.911433 27428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:41.912652 27412 server_base.cc:1034] running on GCE node
I20260923 20:24:41.913223 27412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:41.914143 27412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:41.915369 27412 hybrid_clock.cc:648] HybridClock initialized: now 1790195081915320 us; error 42 us; skew 500 ppm
I20260923 20:24:41.915678 27412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:41 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:41.918825 27412 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:24:41.920254 27412 webserver.cc:460] Webserver started at http://127.14.127.194:37107/ using document root <none> and password file <none>
I20260923 20:24:41.920868 27412 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:41.920962 27412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:41.921201 27412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:24:41 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:41.938689 27412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "a7b8aaf262734381ac7bac3bea41a750"
format_stamp: "Formatted at 2026-09-23 20:24:41 on dist-test-slave-wmd0"
server_key: "1366eca67cc5d66ac138bf783f7ddc72"
server_key_iv: "065c2f05acd1ab5a9fe45e23a7f9fb3d"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:41.939791 27412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "a7b8aaf262734381ac7bac3bea41a750"
format_stamp: "Formatted at 2026-09-23 20:24:41 on dist-test-slave-wmd0"
server_key: "1366eca67cc5d66ac138bf783f7ddc72"
server_key_iv: "065c2f05acd1ab5a9fe45e23a7f9fb3d"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:41.947332 27412 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260923 20:24:41.964383 27435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:41.965988 27412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:24:41.966164 27412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "a7b8aaf262734381ac7bac3bea41a750"
format_stamp: "Formatted at 2026-09-23 20:24:41 on dist-test-slave-wmd0"
server_key: "1366eca67cc5d66ac138bf783f7ddc72"
server_key_iv: "065c2f05acd1ab5a9fe45e23a7f9fb3d"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:41.966398 27412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:41.990744 27412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:41.995584 27412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:41.995908 27412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:41.997001 27412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:41.998500 27412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:41.998577 27412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:41.998736 27412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:41.998790 27412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:42.046826 27412 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:40417
I20260923 20:24:42.046861 27547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:40417 every 8 connection(s)
I20260923 20:24:42.048148 27412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:24:42.054584 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 27412
I20260923 20:24:42.054768 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:24:42.055249 14847 external_mini_cluster.cc:1428] Setting key 394cc68c56effc40eb1295521557f658
Sep 23 20:24:42 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:42.075246 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:42.051066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:48751 (local address 127.14.127.254:37217)
0923 20:24:42.051598 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:42.051604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:42.052674 (+ 1070us) server_negotiation.cc:407] Connection header received
0923 20:24:42.054464 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:42.054470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:42.054558 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:42.054725 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:42.056396 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.057308 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.064695 (+ 7387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.065118 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.065271 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:42.068991 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:42.068998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:42.069008 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:42.069041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:42.072110 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.072867 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.072872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.072876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.073000 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.073432 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.073437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.073441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.073736 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:42.073927 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:42.074636 (+ 709us) server_negotiation.cc:299] Negotiation successful
0923 20:24:42.074871 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":257,"threads_started":1}
I20260923 20:24:42.076346 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:42.051669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:37217 (local address 127.14.127.194:48751)
0923 20:24:42.052496 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:42.052543 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:42.054184 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:42.054971 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:42.054979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:42.055454 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:42.056169 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:42.056190 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.062795 (+ 6605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.062804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:42.064507 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:42.064523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.065396 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.065403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:42.065654 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:42.066364 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:42.066393 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:42.068741 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:42.072261 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:42.072270 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:42.072297 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:42.072719 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:42.073114 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:42.073119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:42.073123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:42.073312 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:42.073887 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:42.073897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:42.074242 (+ 345us) client_negotiation.cc:769] Sending connection context
0923 20:24:42.074539 (+ 297us) client_negotiation.cc:240] Negotiation successful
0923 20:24:42.075079 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":358,"threads_started":1}
I20260923 20:24:42.077840 27548 heartbeater.cc:347] Connected to a master server at 127.14.127.254:37217
I20260923 20:24:42.078162 27548 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:42.078938 27548 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
I20260923 20:24:42.080377 27191 ts_manager.cc:194] Registered new tserver with Master: a7b8aaf262734381ac7bac3bea41a750 (127.14.127.194:40417)
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
I20260923 20:24:42.081203 27191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:48751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:24:42.114077 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:38939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:37217
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:24:42.461911 27405 heartbeater.cc:503] Master 127.14.127.254:37217 was elected leader, sending a full tablet report...
W20260923 20:24:42.474905 27554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:42.475293 27554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:42.475459 27554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:42.483068 27554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:42.483246 27554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:42.483424 27554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:24:42.493423 27554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.127.212:38939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:42.494882 27554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:42.496215 27554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:42.510146 27559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:42.510056 27562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:42.510118 27560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:42.510979 27554 server_base.cc:1034] running on GCE node
I20260923 20:24:42.511662 27554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:42.512693 27554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:42.514036 27554 hybrid_clock.cc:648] HybridClock initialized: now 1790195082513966 us; error 58 us; skew 500 ppm
I20260923 20:24:42.514420 27554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:42 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:42.517647 27554 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:24:42.519201 27554 webserver.cc:460] Webserver started at http://127.14.127.195:46487/ using document root <none> and password file <none>
I20260923 20:24:42.519941 27554 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:42.520093 27554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:42.520427 27554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:24:42 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:42.534909 27554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "de74b22b18324058aebfb0777c633e3d"
format_stamp: "Formatted at 2026-09-23 20:24:42 on dist-test-slave-wmd0"
server_key: "1c38f9f1332cf623fddc419a254d021c"
server_key_iv: "8bc28bbfdf2bba4939e7e19b75d69d2e"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:42.536033 27554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "de74b22b18324058aebfb0777c633e3d"
format_stamp: "Formatted at 2026-09-23 20:24:42 on dist-test-slave-wmd0"
server_key: "1c38f9f1332cf623fddc419a254d021c"
server_key_iv: "8bc28bbfdf2bba4939e7e19b75d69d2e"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:42.542407 27554 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:24:42.557626 27569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:42.559327 27554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:24:42.559468 27554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "de74b22b18324058aebfb0777c633e3d"
format_stamp: "Formatted at 2026-09-23 20:24:42 on dist-test-slave-wmd0"
server_key: "1c38f9f1332cf623fddc419a254d021c"
server_key_iv: "8bc28bbfdf2bba4939e7e19b75d69d2e"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:42.559654 27554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:42.576998 27554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:42.580572 27554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:42.580821 27554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:42.581660 27554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:42.582974 27554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:42.583045 27554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:42.583127 27554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:42.583170 27554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:42.615638 27554 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:36171
I20260923 20:24:42.615676 27681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:36171 every 8 connection(s)
I20260923 20:24:42.616989 27554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:24:42.620536 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 27554
I20260923 20:24:42.620745 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:24:42.621243 14847 external_mini_cluster.cc:1428] Setting key 3612d3db1906dc09d7f66bb00f672836
Sep 23 20:24:42 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:42.636699 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:42.619628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:60251 (local address 127.14.127.254:37217)
0923 20:24:42.620068 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:42.620073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:42.621257 (+ 1184us) server_negotiation.cc:407] Connection header received
0923 20:24:42.623475 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:42.623482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:42.623581 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:42.623698 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:42.625095 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.625817 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.627476 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.627897 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.627990 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:42.631174 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:42.631178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:42.631182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:42.631217 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:42.633562 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.634214 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.634218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.634220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.634357 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.634776 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.634781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.634784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.635065 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:42.635356 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:42.636061 (+ 705us) server_negotiation.cc:299] Negotiation successful
0923 20:24:42.636325 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":209,"threads_started":1}
I20260923 20:24:42.637512 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:42.620265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:37217 (local address 127.14.127.195:60251)
0923 20:24:42.621058 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:42.621106 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:42.623159 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:42.623890 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:42.623899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:42.624389 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:42.624942 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:42.624959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.625951 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.625955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:42.627351 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:42.627363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.628074 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.628078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:42.628263 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:42.628971 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:42.628992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:42.630983 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:42.633693 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:42.633699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:42.633710 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:42.634105 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:42.634449 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:42.634452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:42.634454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:42.634606 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:42.635327 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:42.635333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:42.635700 (+ 367us) client_negotiation.cc:769] Sending connection context
0923 20:24:42.635924 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 20:24:42.636365 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":342,"threads_started":1}
I20260923 20:24:42.638825 27682 heartbeater.cc:347] Connected to a master server at 127.14.127.254:37217
I20260923 20:24:42.639127 27682 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:42.639882 27682 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
I20260923 20:24:42.641435 27191 ts_manager.cc:194] Registered new tserver with Master: de74b22b18324058aebfb0777c633e3d (127.14.127.195:36171)
I20260923 20:24:42.642441 27191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:60251
I20260923 20:24:42.648957 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:24:42.650805 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 27272
Sep 23 20:24:42 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:24:42.667336 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:42.651679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.14.127.193:40079)
0923 20:24:42.652294 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:42.652300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:42.652324 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:24:42.652384 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:42.652390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:42.652577 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:42.652702 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:42.654087 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.655185 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.656631 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.657269 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.657373 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:42.661649 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:42.661661 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:42.661675 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:42.661708 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:42.664899 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.665528 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.665535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.665537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.665669 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.665984 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.665987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.665989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.666314 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:42.666535 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:42.666836 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:24:42.667024 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":329,"threads_started":1}
I20260923 20:24:42.668643 27400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:42.750837 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 27272
I20260923 20:24:42.766892 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 27412
Sep 23 20:24:42 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:24:42.783244 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:42.767850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.14.127.194:40417)
0923 20:24:42.768436 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:42.768443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:42.768468 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:24:42.768528 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:42.768534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:42.768765 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:42.768896 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:42.770126 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.771246 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.772560 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.773095 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.773189 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:42.777454 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:42.777465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:42.777478 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:42.777511 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:42.780546 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.781244 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.781251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.781253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.781372 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.781716 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.781719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.781721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.782062 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:42.782296 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:42.782562 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:24:42.782962 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":320,"threads_started":1}
I20260923 20:24:42.784480 27543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:42.866933 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 27412
I20260923 20:24:42.882697 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 27554
Sep 23 20:24:42 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:24:42.899094 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:42.883709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.14.127.195:36171)
0923 20:24:42.884338 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:42.884345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:42.884372 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:24:42.884428 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:42.884434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:42.884648 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:42.884826 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:42.886184 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.887329 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.888733 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:42.889274 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:42.889453 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:42.893924 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:42.893935 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:42.893948 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:42.894002 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:42.896739 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.897340 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.897346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.897349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.897464 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:42.897781 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:42.897784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:42.897786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:42.898104 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:42.898343 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:42.898576 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 20:24:42.898827 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":312,"threads_started":1}
I20260923 20:24:42.900331 27677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:42.982767 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 27554
I20260923 20:24:42.998507 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 27159
I20260923 20:24:42.999835 27221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:43.106703 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 27159
I20260923 20:24:43.131074 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37217
--webserver_interface=127.14.127.254
--webserver_port=34421
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:37217
--ranger_config_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:38939/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.127.212:38939/kms with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:43.451632 27694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:43.452025 27694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:43.452194 27694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:43.459357 27694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:24:43.459518 27694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:43.459631 27694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:43.459739 27694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:24:43.459833 27694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:24:43.469754 27694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.127.212:38939/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:37217
--ranger_config_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=34421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:43.471103 27694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:43.472436 27694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:43.483942 27702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:43.484032 27699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:43.484022 27700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:43.485110 27694 server_base.cc:1034] running on GCE node
I20260923 20:24:43.485934 27694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:43.487388 27694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:43.488620 27694 hybrid_clock.cc:648] HybridClock initialized: now 1790195083488581 us; error 59 us; skew 500 ppm
I20260923 20:24:43.488965 27694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:43 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195083, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:43.492125 27694 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:24:43.493597 27694 webserver.cc:460] Webserver started at http://127.14.127.254:34421/ using document root <none> and password file <none>
I20260923 20:24:43.494230 27694 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:43.494386 27694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:24:43 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195083, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:43.548563 27694 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:24:43.551642 27710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:43.553187 27694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:24:43.553409 27694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "d358b507c890404e959d730b3a3f7768"
format_stamp: "Formatted at 2026-09-23 20:24:39 on dist-test-slave-wmd0"
server_key: "1c25cfa0491cc84f77f031bd8fd47216"
server_key_iv: "aef1ef43e14c40b47999593512b6f4ab"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:43.554388 27694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:43.570487 27694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:43.574229 27694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:43.574528 27694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:43.593539 27694 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:37217
I20260923 20:24:43.593565 27761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:37217 every 8 connection(s)
I20260923 20:24:43.595111 27694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:24:43.601696 27762 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:24:43.603798 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 27694
I20260923 20:24:43.604801 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:38939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:40079
--local_ip_for_outbound_sockets=127.14.127.193
--tserver_master_addrs=127.14.127.254:37217
--webserver_port=36367
--webserver_interface=127.14.127.193
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.127.212:38939/kms with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:24:43.605834 27762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Bootstrap starting.
I20260923 20:24:43.626267 27762 log.cc:824] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:43.635106 27762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:24:43.635636 27762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Bootstrap complete.
I20260923 20:24:43.644755 27762 raft_consensus.cc:348] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } }
I20260923 20:24:43.645576 27762 raft_consensus.cc:729] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d358b507c890404e959d730b3a3f7768, State: Initialized, Role: FOLLOWER
I20260923 20:24:43.646294 27762 consensus_queue.cc:260] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } }
I20260923 20:24:43.646524 27762 raft_consensus.cc:388] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:24:43.646656 27762 raft_consensus.cc:482] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:24:43.646862 27762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:24:43.650574 27762 raft_consensus.cc:504] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } }
I20260923 20:24:43.651136 27762 leader_election.cc:302] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d358b507c890404e959d730b3a3f7768; no voters:
I20260923 20:24:43.651818 27762 leader_election.cc:288] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:24:43.651961 27766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:24:43.653252 27766 raft_consensus.cc:686] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [term 2 LEADER]: Becoming Leader. State: Replica: d358b507c890404e959d730b3a3f7768, State: Running, Role: LEADER
I20260923 20:24:43.653872 27766 consensus_queue.cc:237] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } }
I20260923 20:24:43.654044 27762 sys_catalog.cc:564] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:24:43.656742 27768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d358b507c890404e959d730b3a3f7768. Latest consensus state: current_term: 2 leader_uuid: "d358b507c890404e959d730b3a3f7768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } } }
I20260923 20:24:43.656790 27767 sys_catalog.cc:455] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d358b507c890404e959d730b3a3f7768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d358b507c890404e959d730b3a3f7768" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37217 } } }
I20260923 20:24:43.657168 27768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:43.657202 27767 sys_catalog.cc:458] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:43.664980 27773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:24:43.670913 27773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:24:43.672341 27773 catalog_manager.cc:1259] Loaded cluster ID: 219278164f274a84817b97b2b2125a7e
I20260923 20:24:43.672495 27773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:24:43.676507 27773 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:24:43.677992 27773 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Loaded TSK: 0
I20260923 20:24:43.679365 27762 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:24:43.679558 27773 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:24:44.020813 27764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:44.021260 27764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:44.021450 27764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:44.033672 27764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:44.033941 27764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:44.034199 27764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:24:44.050954 27764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.127.212:38939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:40079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=36367
--tserver_master_addrs=127.14.127.254:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:44.052942 27764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:44.054777 27764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:44.074229 27800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:44.074335 27799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:44.088340 27803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:44.089367 27764 server_base.cc:1034] running on GCE node
I20260923 20:24:44.090400 27764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:44.094925 27764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:44.096366 27764 hybrid_clock.cc:648] HybridClock initialized: now 1790195084096283 us; error 74 us; skew 500 ppm
I20260923 20:24:44.096792 27764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:44 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:44.105065 27764 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:24:44.107410 27764 webserver.cc:460] Webserver started at http://127.14.127.193:36367/ using document root <none> and password file <none>
I20260923 20:24:44.108465 27764 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:44.108707 27764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:24:44 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:44.156200 27764 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.016s sys 0.004s
I20260923 20:24:44.175274 27810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:44.177399 27764 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.007s sys 0.012s
I20260923 20:24:44.177693 27764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "36c3b8bec27945ef88973234de559a3e"
format_stamp: "Formatted at 2026-09-23 20:24:40 on dist-test-slave-wmd0"
server_key: "c9fcdfe9f0b100359cf9d9f759ce5328"
server_key_iv: "4490a88f50a0b104f26f1b1d446d94e5"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:44.179338 27764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:44.224483 27764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:44.229889 27764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:44.230250 27764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:44.237674 27764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:44.239493 27764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:44.239622 27764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:44.239744 27764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:44.239826 27764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:44.342190 27764 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:40079
I20260923 20:24:44.342219 27923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:40079 every 8 connection(s)
I20260923 20:24:44.344120 27764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:24:44.349555 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 27764
I20260923 20:24:44.350530 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:38939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:40417
--local_ip_for_outbound_sockets=127.14.127.194
--tserver_master_addrs=127.14.127.254:37217
--webserver_port=37107
--webserver_interface=127.14.127.194
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.127.212:38939/kms with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:24:44 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:44.384624 27926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:44.347758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:52701 (local address 127.14.127.254:37217)
0923 20:24:44.348731 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:44.348751 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:44.351866 (+ 3115us) server_negotiation.cc:407] Connection header received
0923 20:24:44.358496 (+ 6630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:44.358521 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:44.359122 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:44.359589 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:44.362171 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:44.364364 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:44.366974 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:44.368260 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:44.368470 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:44.372833 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:44.372846 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:44.372867 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:44.372918 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:44.377087 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:44.378114 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:44.378124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:44.378132 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:44.378360 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:44.378942 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:44.378948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:44.378951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:44.379624 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:44.379833 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:44.382953 (+ 3120us) server_negotiation.cc:299] Negotiation successful
0923 20:24:44.383383 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":678,"threads_started":1}
I20260923 20:24:44.385174 27927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:44.347748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:37217 (local address 127.14.127.193:52701)
0923 20:24:44.351650 (+ 3902us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:44.351708 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:44.357572 (+ 5864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:44.360453 (+ 2881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:44.360463 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:44.361105 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:44.361905 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:44.361935 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:44.364641 (+ 2706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:44.364647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:44.366805 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:44.366823 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:44.368541 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:44.368548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:44.368833 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:44.369655 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:44.369693 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:44.372466 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:44.377289 (+ 4823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:44.377297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:44.377313 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:44.377918 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:44.378517 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:44.378523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:44.378527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:44.378795 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:44.379925 (+ 1130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:44.379934 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:44.380368 (+ 434us) client_negotiation.cc:769] Sending connection context
0923 20:24:44.380689 (+ 321us) client_negotiation.cc:240] Negotiation successful
0923 20:24:44.381271 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3600,"thread_start_us":2257,"threads_started":1}
I20260923 20:24:44.389487 27924 heartbeater.cc:347] Connected to a master server at 127.14.127.254:37217
I20260923 20:24:44.389811 27924 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:44.390770 27924 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
W20260923 20:24:44.394029 27924 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:37217 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:24:44.394304 27924 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:44.394591 27924 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
I20260923 20:24:44.395953 27924 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:44.396219 27924 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
W20260923 20:24:44.397418 27924 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:24:44.770057 27928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:44.770465 27928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:44.770718 27928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:44.779028 27928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:44.779228 27928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:44.779475 27928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:24:44.795346 27928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.127.212:38939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:40417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=37107
--tserver_master_addrs=127.14.127.254:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:44.797202 27928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:44.799034 27928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:44.814477 27944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:44.814436 27943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:44.815661 27947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:44.817080 27928 server_base.cc:1034] running on GCE node
I20260923 20:24:44.817914 27928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:44.819924 27928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:44.823505 27928 hybrid_clock.cc:648] HybridClock initialized: now 1790195084822160 us; error 1348 us; skew 500 ppm
I20260923 20:24:44.823942 27928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:44 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:44.828327 27928 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:24:44.830392 27928 webserver.cc:460] Webserver started at http://127.14.127.194:37107/ using document root <none> and password file <none>
I20260923 20:24:44.831331 27928 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:44.831542 27928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:24:44 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:44.880858 27928 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.002s sys 0.015s
I20260923 20:24:44.895358 27955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:44.900213 27928 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.012s
I20260923 20:24:44.900485 27928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "a7b8aaf262734381ac7bac3bea41a750"
format_stamp: "Formatted at 2026-09-23 20:24:41 on dist-test-slave-wmd0"
server_key: "1366eca67cc5d66ac138bf783f7ddc72"
server_key_iv: "065c2f05acd1ab5a9fe45e23a7f9fb3d"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:44.902060 27928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 23 20:24:44 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:44.980996 27928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:44.985986 27928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:44.986300 27928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:44.987457 27928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:44.988900 27928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:44.989023 27928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:44.989161 27928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:44.989248 27928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:45.053515 27928 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:40417
I20260923 20:24:45.053598 28068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:40417 every 8 connection(s)
I20260923 20:24:45.055380 27928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:24:45.062457 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 27928
I20260923 20:24:45.063323 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.127.212:38939/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:36171
--local_ip_for_outbound_sockets=127.14.127.195
--tserver_master_addrs=127.14.127.254:37217
--webserver_port=46487
--webserver_interface=127.14.127.195
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.127.212:38939/kms with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:24:45 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:45.117695 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:45.064075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:37217 (local address 127.14.127.194:45625)
0923 20:24:45.083678 (+ 19603us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:45.083741 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:45.094709 (+ 10968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:45.096094 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:45.096109 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:45.096863 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:45.097834 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:45.097863 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:45.100269 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:45.100278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:45.102756 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:45.102775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:45.104256 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:45.104265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:45.104634 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:45.105509 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:45.105555 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:45.108750 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:45.112919 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:45.112928 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:45.112947 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:45.113522 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:45.113967 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:45.113972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:45.113976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:45.114212 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:45.114923 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:45.114931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:45.115379 (+ 448us) client_negotiation.cc:769] Sending connection context
0923 20:24:45.115672 (+ 293us) client_negotiation.cc:240] Negotiation successful
0923 20:24:45.116239 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19255,"thread_start_us":18947,"threads_started":1}
I20260923 20:24:45.118830 28072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:45.064929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:45625 (local address 127.14.127.254:37217)
0923 20:24:45.083014 (+ 18085us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:45.083021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:45.086764 (+ 3743us) server_negotiation.cc:407] Connection header received
0923 20:24:45.095167 (+ 8403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:45.095174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:45.095295 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:45.095427 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:45.098731 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:45.100036 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:45.102964 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:45.103734 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:45.103880 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:45.109067 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:45.109090 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:45.109099 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:45.109149 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:45.112748 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:45.113693 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:45.113699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:45.113703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:45.113846 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:45.114337 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:45.114343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:45.114347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:45.114719 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:45.114853 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:45.118204 (+ 3351us) server_negotiation.cc:299] Negotiation successful
0923 20:24:45.118437 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17930,"thread_start_us":17797,"threads_started":1}
I20260923 20:24:45.120858 28069 heartbeater.cc:347] Connected to a master server at 127.14.127.254:37217
I20260923 20:24:45.121200 28069 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:45.122103 28069 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
I20260923 20:24:45.124084 27727 ts_manager.cc:194] Registered new tserver with Master: a7b8aaf262734381ac7bac3bea41a750 (127.14.127.194:40417)
I20260923 20:24:45.125625 27727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:45625
Sep 23 20:24:45 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:45.397572 27924 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:45.398056 27924 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
I20260923 20:24:45.399474 27727 ts_manager.cc:194] Registered new tserver with Master: 36c3b8bec27945ef88973234de559a3e (127.14.127.193:40079)
I20260923 20:24:45.400758 27727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:52701
W20260923 20:24:45.527269 28073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:45.527652 28073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:45.527834 28073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:45.538880 28073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:45.539021 28073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:45.539249 28073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:24:45.554457 28073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.127.212:38939/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:36171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=46487
--tserver_master_addrs=127.14.127.254:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:45.556298 28073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:45.557948 28073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:45.574216 28078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:45.574936 28079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:45.583107 28073 server_base.cc:1034] running on GCE node
W20260923 20:24:45.583855 28081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:45.584721 28073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:45.586149 28073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:45.587409 28073 hybrid_clock.cc:648] HybridClock initialized: now 1790195085587352 us; error 50 us; skew 500 ppm
I20260923 20:24:45.587767 28073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:45 dist-test-slave-wmd0 krb5kdc[26715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:45.591626 28073 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:24:45.620599 28073 webserver.cc:460] Webserver started at http://127.14.127.195:46487/ using document root <none> and password file <none>
I20260923 20:24:45.621665 28073 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:45.621842 28073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:24:45 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
Sep 23 20:24:45 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195084, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for HTTP/127.14.127.212@KRBTEST.COM
I20260923 20:24:45.660439 28073 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 20:24:45.668155 28090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:45.670396 28073 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.001s
I20260923 20:24:45.670642 28073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "de74b22b18324058aebfb0777c633e3d"
format_stamp: "Formatted at 2026-09-23 20:24:42 on dist-test-slave-wmd0"
server_key: "1c38f9f1332cf623fddc419a254d021c"
server_key_iv: "8bc28bbfdf2bba4939e7e19b75d69d2e"
server_key_version: "kuduclusterkey@0"
I20260923 20:24:45.671736 28073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:45.743108 28073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:45.752905 28073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:45.753264 28073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:45.768221 28073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:45.782888 28073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:45.783030 28073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:45.783222 28073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:45.783349 28073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:46.008607 28073 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:36171
I20260923 20:24:46.008651 28209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:36171 every 8 connection(s)
I20260923 20:24:46.010445 28073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:24:46.017611 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 28073
Sep 23 20:24:46 dist-test-slave-wmd0 krb5kdc[26715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195085, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:46.045270 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.013167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:48097 (local address 127.14.127.254:37217)
0923 20:24:46.013635 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.013641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.014874 (+ 1233us) server_negotiation.cc:407] Connection header received
0923 20:24:46.017958 (+ 3084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.017966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.018068 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.018162 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:46.029210 (+ 11048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.030786 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.033332 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.033756 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.033902 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.038205 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:46.038214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:46.038222 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:46.038298 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:46.041251 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.042253 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.042259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.042263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.042424 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.043001 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.043008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.043012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.043386 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:46.043550 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.044605 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.044857 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260923 20:24:46.046303 28213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.013753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:37217 (local address 127.14.127.195:48097)
0923 20:24:46.014696 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:46.014752 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:46.017589 (+ 2837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:46.018776 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:46.018790 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:46.025758 (+ 6968us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:46.026916 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.026952 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.031091 (+ 4139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.031100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.033170 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.033182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.034013 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.034020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.034264 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.035811 (+ 1547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:46.035834 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:46.037968 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:46.041396 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:46.041406 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:46.041424 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:46.042082 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:46.042567 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:46.042572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:46.042576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:46.042849 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:46.043635 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:46.043647 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:46.044140 (+ 493us) client_negotiation.cc:769] Sending connection context
0923 20:24:46.044465 (+ 325us) client_negotiation.cc:240] Negotiation successful
0923 20:24:46.045059 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":355,"threads_started":1}
I20260923 20:24:46.047940 28210 heartbeater.cc:347] Connected to a master server at 127.14.127.254:37217
I20260923 20:24:46.048306 28210 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:46.049157 28214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.019286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.14.127.254:37217)
0923 20:24:46.031730 (+ 12444us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.031736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.032348 (+ 612us) server_negotiation.cc:407] Connection header received
0923 20:24:46.032515 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.032521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.032606 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.032719 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:46.034752 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.035594 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.037519 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.038184 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.038298 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.041635 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:46.041641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:46.041646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:46.041688 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:46.045058 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.046332 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.046337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.046341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.046479 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.047466 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.047471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.047474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.047790 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:46.048117 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.048418 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.048631 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12317,"thread_start_us":12342,"threads_started":1}
I20260923 20:24:46.049167 28210 heartbeater.cc:511] Master 127.14.127.254:37217 requested a full tablet report, sending...
I20260923 20:24:46.051215 27726 ts_manager.cc:194] Registered new tserver with Master: de74b22b18324058aebfb0777c633e3d (127.14.127.195:36171)
I20260923 20:24:46.052186 27726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:48097
I20260923 20:24:46.065524 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:24:46.083536 28214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.069057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.14.127.254:37217)
0923 20:24:46.069256 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.069261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.069722 (+ 461us) server_negotiation.cc:407] Connection header received
0923 20:24:46.069848 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.069853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.069940 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.070045 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:46.071751 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.072830 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.074106 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.074802 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.074923 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.077424 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:46.077431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:46.077436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:46.077476 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:46.080813 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.081537 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.081542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.081546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.081718 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.082082 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.082087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.082091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.082446 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:46.082715 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.082986 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.083217 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:24:46.088675 27726 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:24:46.096648 27726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:24:46.129928 28069 heartbeater.cc:503] Master 127.14.127.254:37217 was elected leader, sending a full tablet report...
I20260923 20:24:46.136101 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.112609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.14.127.193:40079)
0923 20:24:46.128044 (+ 15435us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.128052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.128080 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:24:46.128183 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.128191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.128435 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.129255 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:46.129429 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.131008 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.133370 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.134886 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.135094 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.135279 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.135570 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.135736 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15254,"thread_start_us":14758,"threads_started":1}
I20260923 20:24:46.136174 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.113016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:40079 (local address 127.0.0.1:35500)
0923 20:24:46.120355 (+ 7339us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:46.120378 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:46.120519 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:46.128679 (+ 8160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:46.128686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:46.128738 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:46.129047 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.129056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.131170 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.131177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.133159 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.133175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.135126 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.135131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.135327 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.135377 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 20:24:46.135518 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 20:24:46.135745 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7188,"mutex_wait_us":26,"thread_start_us":13803,"threads_started":1}
I20260923 20:24:46.137641 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.117668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:40417 (local address 127.0.0.1:47042)
0923 20:24:46.127119 (+ 9451us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:46.127140 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:46.127256 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:46.130116 (+ 2860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:46.130120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:46.130144 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:46.130558 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.130578 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.132453 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.132459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.134691 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.134710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.136811 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.136817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.136986 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.137018 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:24:46.137106 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 20:24:46.137245 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9292,"mutex_wait_us":24,"thread_start_us":2390,"threads_started":1}
I20260923 20:24:46.137904 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.118029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.14.127.194:40417)
0923 20:24:46.127416 (+ 9387us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.127426 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.127459 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:24:46.127573 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.127582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.127863 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.130818 (+ 2955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:46.131047 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.132267 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.134847 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.136281 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.136469 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.136635 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.137189 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.137363 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9202,"thread_start_us":9065,"threads_started":1}
I20260923 20:24:46.139567 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.112038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:36171 (local address 127.0.0.1:58484)
0923 20:24:46.116110 (+ 4072us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:46.116169 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:46.116377 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:46.129619 (+ 13242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:46.129626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:46.129649 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:46.129961 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.129973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.134026 (+ 4053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.134034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.136167 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.136183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.138745 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.138751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.138915 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.138957 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:24:46.139057 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 20:24:46.139193 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3761,"mutex_wait_us":49,"thread_start_us":3583,"threads_started":1}
I20260923 20:24:46.139612 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.112655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.14.127.195:36171)
0923 20:24:46.129053 (+ 16398us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.129061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.129093 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:24:46.129199 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.129207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.129410 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.129616 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:46.132517 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.133879 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.136368 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.138492 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.138742 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.138905 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.139102 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.139258 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16231,"thread_start_us":17214,"threads_started":1}
I20260923 20:24:46.140671 27859 tablet_service.cc:1467] Processing CreateTablet for tablet a719a3cc132449fea4cdf69afb9a07fa (DEFAULT_TABLE table=test-table [id=029b360410f34ccdac508029f8faf4de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:24:46.141186 28004 tablet_service.cc:1467] Processing CreateTablet for tablet a719a3cc132449fea4cdf69afb9a07fa (DEFAULT_TABLE table=test-table [id=029b360410f34ccdac508029f8faf4de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:24:46.141978 27859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a719a3cc132449fea4cdf69afb9a07fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:46.142378 28004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a719a3cc132449fea4cdf69afb9a07fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:46.145569 28143 tablet_service.cc:1467] Processing CreateTablet for tablet a719a3cc132449fea4cdf69afb9a07fa (DEFAULT_TABLE table=test-table [id=029b360410f34ccdac508029f8faf4de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:24:46.146688 28143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a719a3cc132449fea4cdf69afb9a07fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:46.153924 28231 tablet_bootstrap.cc:492] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: Bootstrap starting.
I20260923 20:24:46.156978 28231 tablet_bootstrap.cc:654] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:46.158241 28231 log.cc:824] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:46.161289 28233 tablet_bootstrap.cc:492] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: Bootstrap starting.
I20260923 20:24:46.161775 28231 tablet_bootstrap.cc:492] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: No bootstrap required, opened a new log
I20260923 20:24:46.162047 28231 ts_tablet_manager.cc:1397] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 20:24:46.163130 28234 tablet_bootstrap.cc:492] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Bootstrap starting.
I20260923 20:24:46.165256 28233 tablet_bootstrap.cc:654] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:46.166134 28234 tablet_bootstrap.cc:654] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:46.166646 28233 log.cc:824] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:46.167467 28234 log.cc:824] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:46.169924 28233 tablet_bootstrap.cc:492] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: No bootstrap required, opened a new log
I20260923 20:24:46.170186 28233 ts_tablet_manager.cc:1397] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 20:24:46.173359 28231 raft_consensus.cc:348] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.173777 28231 raft_consensus.cc:374] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:46.173866 28231 raft_consensus.cc:729] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c3b8bec27945ef88973234de559a3e, State: Initialized, Role: FOLLOWER
I20260923 20:24:46.174490 28231 consensus_queue.cc:260] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.175601 27924 heartbeater.cc:503] Master 127.14.127.254:37217 was elected leader, sending a full tablet report...
I20260923 20:24:46.177026 28234 tablet_bootstrap.cc:492] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: No bootstrap required, opened a new log
I20260923 20:24:46.177314 28234 ts_tablet_manager.cc:1397] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260923 20:24:46.177522 28231 ts_tablet_manager.cc:1428] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260923 20:24:46.179901 28233 raft_consensus.cc:348] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.180322 28233 raft_consensus.cc:374] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:46.180423 28233 raft_consensus.cc:729] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b8aaf262734381ac7bac3bea41a750, State: Initialized, Role: FOLLOWER
I20260923 20:24:46.181046 28233 consensus_queue.cc:260] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.182497 28233 ts_tablet_manager.cc:1428] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 20:24:46.186806 28234 raft_consensus.cc:348] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.187145 28234 raft_consensus.cc:374] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:46.187223 28234 raft_consensus.cc:729] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de74b22b18324058aebfb0777c633e3d, State: Initialized, Role: FOLLOWER
I20260923 20:24:46.187668 28234 consensus_queue.cc:260] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.188588 28210 heartbeater.cc:503] Master 127.14.127.254:37217 was elected leader, sending a full tablet report...
I20260923 20:24:46.189185 28234 ts_tablet_manager.cc:1428] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260923 20:24:46.192039 28237 raft_consensus.cc:482] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:24:46.192293 28237 raft_consensus.cc:504] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.193842 28237 leader_election.cc:288] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de74b22b18324058aebfb0777c633e3d (127.14.127.195:36171), a7b8aaf262734381ac7bac3bea41a750 (127.14.127.194:40417)
I20260923 20:24:46.202172 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.194072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:42979 (local address 127.14.127.195:36171)
0923 20:24:46.194371 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.194377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.194734 (+ 357us) server_negotiation.cc:407] Connection header received
0923 20:24:46.195026 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.195031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.195100 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.195164 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:46.196073 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.196950 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.199826 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.201038 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.201210 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.201314 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.201802 (+ 488us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.201930 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260923 20:24:46.202176 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.194919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:51897 (local address 127.14.127.194:40417)
0923 20:24:46.195123 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.195130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.195390 (+ 260us) server_negotiation.cc:407] Connection header received
0923 20:24:46.195507 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.195514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.195592 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.195686 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:46.196291 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.197128 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.199970 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.201165 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.201308 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.201380 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.201811 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.201931 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:24:46.202442 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.194719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:40417 (local address 127.14.127.193:51897)
0923 20:24:46.195292 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:46.195306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:46.195411 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:46.195702 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:46.195705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:46.195725 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:46.196140 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.196152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.197310 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.197349 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.199616 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.199630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.201364 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.201369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.201584 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.201665 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 20:24:46.201757 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:24:46.201946 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":107,"thread_start_us":294,"threads_started":1}
I20260923 20:24:46.202466 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.193995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:36171 (local address 127.14.127.193:42979)
0923 20:24:46.194592 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:46.194657 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:46.194853 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:46.195335 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:46.195338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:46.195440 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:46.195933 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.195944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.197113 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.197152 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.199617 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:46.199630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.201298 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.201310 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:46.201547 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.201661 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 20:24:46.201757 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 20:24:46.201945 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":121,"spinlock_wait_cycles":33152,"thread_start_us":324,"threads_started":1}
I20260923 20:24:46.203296 28165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a719a3cc132449fea4cdf69afb9a07fa" candidate_uuid: "36c3b8bec27945ef88973234de559a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de74b22b18324058aebfb0777c633e3d" is_pre_election: true
I20260923 20:24:46.203469 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a719a3cc132449fea4cdf69afb9a07fa" candidate_uuid: "36c3b8bec27945ef88973234de559a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7b8aaf262734381ac7bac3bea41a750" is_pre_election: true
I20260923 20:24:46.203817 28165 raft_consensus.cc:2454] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36c3b8bec27945ef88973234de559a3e in term 0.
I20260923 20:24:46.203994 28024 raft_consensus.cc:2454] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36c3b8bec27945ef88973234de559a3e in term 0.
I20260923 20:24:46.204480 27813 leader_election.cc:302] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36c3b8bec27945ef88973234de559a3e, de74b22b18324058aebfb0777c633e3d; no voters:
I20260923 20:24:46.204824 28237 raft_consensus.cc:2781] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:24:46.204943 28237 raft_consensus.cc:482] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:24:46.205060 28237 raft_consensus.cc:3037] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:46.207297 28237 raft_consensus.cc:504] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.207836 28237 leader_election.cc:288] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [CANDIDATE]: Term 1 election: Requested vote from peers de74b22b18324058aebfb0777c633e3d (127.14.127.195:36171), a7b8aaf262734381ac7bac3bea41a750 (127.14.127.194:40417)
I20260923 20:24:46.208319 28024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a719a3cc132449fea4cdf69afb9a07fa" candidate_uuid: "36c3b8bec27945ef88973234de559a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7b8aaf262734381ac7bac3bea41a750"
I20260923 20:24:46.208338 28165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a719a3cc132449fea4cdf69afb9a07fa" candidate_uuid: "36c3b8bec27945ef88973234de559a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de74b22b18324058aebfb0777c633e3d"
I20260923 20:24:46.208536 28165 raft_consensus.cc:3037] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:46.208546 28024 raft_consensus.cc:3037] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:46.210806 28024 raft_consensus.cc:2454] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36c3b8bec27945ef88973234de559a3e in term 1.
I20260923 20:24:46.210839 28165 raft_consensus.cc:2454] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36c3b8bec27945ef88973234de559a3e in term 1.
I20260923 20:24:46.211189 27813 leader_election.cc:302] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36c3b8bec27945ef88973234de559a3e, a7b8aaf262734381ac7bac3bea41a750; no voters:
I20260923 20:24:46.211508 28237 raft_consensus.cc:2781] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:24:46.211942 28237 raft_consensus.cc:686] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 1 LEADER]: Becoming Leader. State: Replica: 36c3b8bec27945ef88973234de559a3e, State: Running, Role: LEADER
I20260923 20:24:46.212383 28237 consensus_queue.cc:237] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } }
I20260923 20:24:46.216861 27727 catalog_manager.cc:5495] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 36c3b8bec27945ef88973234de559a3e (127.14.127.193). New cstate: current_term: 1 leader_uuid: "36c3b8bec27945ef88973234de559a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36c3b8bec27945ef88973234de559a3e" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 } health_report { overall_health: UNKNOWN } } }
I20260923 20:24:46.255975 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.250765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.14.127.193:40079)
0923 20:24:46.250983 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.250989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.251010 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:24:46.251123 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.251127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.251201 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.251351 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:24:46.251736 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.252456 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.253806 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.254234 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.254365 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.254910 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:24:46.255411 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:24:46.255577 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.255626 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.255702 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 20:24:46.262480 28211 tablet.cc:2369] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:24:46.264012 28165 raft_consensus.cc:1261] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 1 FOLLOWER]: Refusing update from remote peer 36c3b8bec27945ef88973234de559a3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:24:46.264050 28024 raft_consensus.cc:1261] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 1 FOLLOWER]: Refusing update from remote peer 36c3b8bec27945ef88973234de559a3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:24:46.264936 28242 consensus_queue.cc:1035] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de74b22b18324058aebfb0777c633e3d" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 36171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:24:46.265376 28237 consensus_queue.cc:1035] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7b8aaf262734381ac7bac3bea41a750" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 40417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:24:46.274910 28245 mvcc.cc:205] Tried to move back new op lower bound from 7332639073327390720 to 7332639073129402368. Current Snapshot: MvccSnapshot[applied={T|T < 7332639073327390720}]
I20260923 20:24:46.275108 28246 mvcc.cc:205] Tried to move back new op lower bound from 7332639073327390720 to 7332639073129402368. Current Snapshot: MvccSnapshot[applied={T|T < 7332639073327390720}]
I20260923 20:24:46.290450 27726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:24:46.290774 27726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:24:46.294339 27726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d358b507c890404e959d730b3a3f7768: Sending DeleteTablet for 3 replicas of tablet a719a3cc132449fea4cdf69afb9a07fa
I20260923 20:24:46.295526 27859 tablet_service.cc:1514] Processing DeleteTablet for tablet a719a3cc132449fea4cdf69afb9a07fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:24:46 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:35500
I20260923 20:24:46.295588 28143 tablet_service.cc:1514] Processing DeleteTablet for tablet a719a3cc132449fea4cdf69afb9a07fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:24:46 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:58484
I20260923 20:24:46.295648 28004 tablet_service.cc:1514] Processing DeleteTablet for tablet a719a3cc132449fea4cdf69afb9a07fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:24:46 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:47042
I20260923 20:24:46.306486 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 27764
I20260923 20:24:46.308065 28255 tablet_replica.cc:330] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: stopping tablet replica
I20260923 20:24:46.308694 28257 tablet_replica.cc:330] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: stopping tablet replica
I20260923 20:24:46.308670 28255 raft_consensus.cc:2229] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:24:46.309119 28257 raft_consensus.cc:2229] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:24:46.309576 28255 raft_consensus.cc:2258] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:24:46.309634 28257 raft_consensus.cc:2258] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:24:46.311902 28257 ts_tablet_manager.cc:1905] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:24:46.312489 28256 tablet_replica.cc:330] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: stopping tablet replica
I20260923 20:24:46.311924 28255 ts_tablet_manager.cc:1905] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:24:46.318888 28257 ts_tablet_manager.cc:1918] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:24:46.319053 28257 log.cc:1196] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/a719a3cc132449fea4cdf69afb9a07fa
I20260923 20:24:46.319591 28257 ts_tablet_manager.cc:1939] T a719a3cc132449fea4cdf69afb9a07fa P a7b8aaf262734381ac7bac3bea41a750: Deleting consensus metadata
I20260923 20:24:46.319823 28256 raft_consensus.cc:2229] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:24:46.320418 28256 raft_consensus.cc:2258] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:24:46.322688 28256 ts_tablet_manager.cc:1905] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:24:46.324846 27713 catalog_manager.cc:4841] TS a7b8aaf262734381ac7bac3bea41a750 (127.14.127.194:40417): tablet a719a3cc132449fea4cdf69afb9a07fa (table test-table [id=029b360410f34ccdac508029f8faf4de]) successfully deleted
I20260923 20:24:46.328029 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.310357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.14.127.193:40079)
0923 20:24:46.310567 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.310574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.310594 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:24:46.310681 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.310685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.310759 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.310862 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:46.312736 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.314005 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.315928 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.316600 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.316727 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.320340 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:46.320356 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:46.320375 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:46.320424 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:46.324600 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.325561 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.325575 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.325579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.325777 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.326225 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.326230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.326234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.326760 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:46.327102 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.327400 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.327653 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:24:46.328986 27919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:46.329223 28256 ts_tablet_manager.cc:1918] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:24:46.329383 28256 log.cc:1196] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/a719a3cc132449fea4cdf69afb9a07fa
I20260923 20:24:46.330066 28256 ts_tablet_manager.cc:1939] T a719a3cc132449fea4cdf69afb9a07fa P de74b22b18324058aebfb0777c633e3d: Deleting consensus metadata
I20260923 20:24:46.332172 27713 catalog_manager.cc:4841] TS de74b22b18324058aebfb0777c633e3d (127.14.127.195:36171): tablet a719a3cc132449fea4cdf69afb9a07fa (table test-table [id=029b360410f34ccdac508029f8faf4de]) successfully deleted
I20260923 20:24:46.428624 28255 ts_tablet_manager.cc:1918] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:24:46.428930 28255 log.cc:1196] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/a719a3cc132449fea4cdf69afb9a07fa
I20260923 20:24:46.429643 28255 ts_tablet_manager.cc:1939] T a719a3cc132449fea4cdf69afb9a07fa P 36c3b8bec27945ef88973234de559a3e: Deleting consensus metadata
I20260923 20:24:46.431690 27712 catalog_manager.cc:4841] TS 36c3b8bec27945ef88973234de559a3e (127.14.127.193:40079): tablet a719a3cc132449fea4cdf69afb9a07fa (table test-table [id=029b360410f34ccdac508029f8faf4de]) successfully deleted
I20260923 20:24:46.439079 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 27764
I20260923 20:24:46.457268 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 27928
I20260923 20:24:46.470603 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.458009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.14.127.194:40417)
0923 20:24:46.458209 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.458217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.458281 (+ 64us) server_negotiation.cc:407] Connection header received
0923 20:24:46.458501 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.458511 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.458573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.458722 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:46.460036 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.460755 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.462004 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.462390 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.462481 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.465197 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:46.465211 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:46.465235 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:46.465269 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:46.468262 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.468879 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.468885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.468888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.469013 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.469352 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.469355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.469357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.469703 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:46.469933 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.470160 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.470326 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:24:46.471345 28064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:46.560379 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 27928
I20260923 20:24:46.577502 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 28073
I20260923 20:24:46.590996 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:46.578323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.14.127.195:36171)
0923 20:24:46.578505 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:46.578509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:46.578546 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:24:46.578746 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:46.578751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:46.578835 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:46.578916 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:46.580217 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.580940 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.582195 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:46.582684 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:46.582826 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:46.585380 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:46.585396 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:46.585411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:46.585444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:46.588151 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.588900 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.588908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.588910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.589032 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:46.589408 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:46.589414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:46.589417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:46.589774 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:46.590085 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:46.590395 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 20:24:46.590643 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:24:46.591836 28205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:46.690438 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 28073
I20260923 20:24:46.707687 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 27694
I20260923 20:24:46.708871 27757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:46.821365 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 27694
I20260923 20:24:46.843257 14847 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 20:24:47.178850 14847 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 20:24:47.179096 14847 mini_ranger.cc:66] Stopping Ranger...
I20260923 20:24:47.524957 14847 mini_ranger.cc:68] Stopped Ranger
2026-09-23 20:24:47.525 UTC [26745] LOG: received smart shutdown request
2026-09-23 20:24:47.528 UTC [26745] LOG: background worker "logical replication launcher" (PID 26754) exited with exit code 1
2026-09-23 20:24:47.529 UTC [26749] LOG: shutting down
2026-09-23 20:24:47.696 UTC [26745] LOG: database system is shut down
2026-09-23T20:24:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54092 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:24:47 dist-test-slave-wmd0 krb5kdc[28270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:24:47 dist-test-slave-wmd0 krb5kdc[28270](info): set up 2 sockets
Sep 23 20:24:47 dist-test-slave-wmd0 krb5kdc[28270](info): commencing operation
krb5kdc: starting...
W20260923 20:24:48.927079 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:24:49 dist-test-slave-wmd0 krb5kdc[28270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:24:49.157797 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:42891
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:42891,127.14.127.253:45141,127.14.127.252:45899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:49.486863 28283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:49.487224 28283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:49.487381 28283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:49.494670 28283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:24:49.494827 28283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:49.494921 28283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:49.495024 28283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:24:49.495112 28283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:24:49.504945 28283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:42891,127.14.127.253:45141,127.14.127.252:45899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:42891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:49.506364 28283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:49.507761 28283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:49.519894 28288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:49.519840 28291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:49.519862 28289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:49.520958 28283 server_base.cc:1034] running on GCE node
I20260923 20:24:49.521778 28283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:49.523312 28283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:49.524544 28283 hybrid_clock.cc:648] HybridClock initialized: now 1790195089524510 us; error 51 us; skew 500 ppm
I20260923 20:24:49.524909 28283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:49 dist-test-slave-wmd0 krb5kdc[28270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195089, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:49.534029 28283 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:24:49.535629 28283 webserver.cc:460] Webserver started at http://127.14.127.254:46557/ using document root <none> and password file <none>
I20260923 20:24:49.536369 28283 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:49.536520 28283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:49.536834 28283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:24:49.539995 28283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "a9c274fc195d4502ba9df761ac7376a5"
format_stamp: "Formatted at 2026-09-23 20:24:49 on dist-test-slave-wmd0"
server_key: "62e70061296b363febe945913044a4b8"
server_key_iv: "4e1c00ae4f8560da3a9fd760ccafa8e6"
server_key_version: "encryptionkey@0"
I20260923 20:24:49.540805 28283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "a9c274fc195d4502ba9df761ac7376a5"
format_stamp: "Formatted at 2026-09-23 20:24:49 on dist-test-slave-wmd0"
server_key: "62e70061296b363febe945913044a4b8"
server_key_iv: "4e1c00ae4f8560da3a9fd760ccafa8e6"
server_key_version: "encryptionkey@0"
I20260923 20:24:49.545975 28283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:24:49.549487 28298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:49.550915 28283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:24:49.551101 28283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "a9c274fc195d4502ba9df761ac7376a5"
format_stamp: "Formatted at 2026-09-23 20:24:49 on dist-test-slave-wmd0"
server_key: "62e70061296b363febe945913044a4b8"
server_key_iv: "4e1c00ae4f8560da3a9fd760ccafa8e6"
server_key_version: "encryptionkey@0"
I20260923 20:24:49.551308 28283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:49.563512 28283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:49.568039 28283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:49.568328 28283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:49.587284 28283 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:42891
I20260923 20:24:49.587313 28349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:42891 every 8 connection(s)
I20260923 20:24:49.588641 28283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:24:49.592260 28350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:49.598882 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 28283
I20260923 20:24:49.599121 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:24:49.599916 14847 external_mini_cluster.cc:1428] Setting key 48cd2a4b03411c15c1c36fbb1a6e8e92
I20260923 20:24:49.598876 28350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:24:49 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195089, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.127.253@KRBTEST.COM; defaulting to no policy
I20260923 20:24:49.624315 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.602848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42891 (local address 127.0.0.1:52124)
0923 20:24:49.603690 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.603750 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:49.605897 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:49.606738 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:49.606746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:49.607531 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:49.608043 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:49.608070 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:49.610455 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:49.610459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:49.611591 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:49.611603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:49.612933 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:49.612938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:49.613130 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:49.613805 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:49.613828 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:49.616349 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:49.620158 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:49.620174 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:49.620188 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:49.620799 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:49.621205 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:49.621212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:49.621217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:49.621456 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:49.622304 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:49.622309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:49.622505 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 20:24:49.622793 (+ 288us) client_negotiation.cc:240] Negotiation successful
0923 20:24:49.623201 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":52,"thread_start_us":381,"threads_started":1}
I20260923 20:24:49.624378 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.602849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.14.127.254:42891)
0923 20:24:49.604525 (+ 1676us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:49.604533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:49.604581 (+ 48us) server_negotiation.cc:407] Connection header received
0923 20:24:49.606268 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:49.606274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:49.606497 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:49.606736 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:49.608281 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:49.610279 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:49.611739 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:49.612703 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:49.612915 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:49.616592 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:49.616605 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:49.616623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:49.616667 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:49.620009 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:49.620930 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:49.620936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:49.620939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:49.621085 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:49.621568 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:49.621571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:49.621573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:49.622164 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:49.622308 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:49.622981 (+ 673us) server_negotiation.cc:299] Negotiation successful
0923 20:24:49.623318 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":1456,"thread_start_us":1252,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.253@KRBTEST.COM" created.
I20260923 20:24:49.627496 28350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:24:49.628695 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.628324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49460)
0923 20:24:49.628485 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.628587 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 20:24:49.629106 28302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
I20260923 20:24:49.630713 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.630335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49474)
0923 20:24:49.630467 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.630532 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:24:49.631242 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45141: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab.
Entry for principal kudu/127.14.127.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab.
I20260923 20:24:49.653499 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.253:45141
--webserver_interface=127.14.127.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab
--principal=kudu/127.14.127.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:42891,127.14.127.253:45141,127.14.127.252:45899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:24:49.680675 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } attempt: 1
I20260923 20:24:49.681876 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.681419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49490)
0923 20:24:49.681618 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.681744 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:24:49.682952 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.682568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49496)
0923 20:24:49.682757 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.682828 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 20:24:49.683149 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45141: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
I20260923 20:24:49.751484 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } attempt: 2
I20260923 20:24:49.752679 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.752199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49504)
0923 20:24:49.752457 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.752545 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:24:49.753664 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.753324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49508)
0923 20:24:49.753465 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.753549 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:24:49.753921 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45141: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
I20260923 20:24:49.845259 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } attempt: 3
I20260923 20:24:49.846448 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.846012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49512)
0923 20:24:49.846212 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.846321 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:24:49.847563 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.847248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49526)
0923 20:24:49.847375 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.847440 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:24:49.847807 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45141: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
W20260923 20:24:49.976416 28357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:49.976708 28357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:49.976796 28357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:49.984630 28357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:24:49.984740 28357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:49.984786 28357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:49.984843 28357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:24:49.984881 28357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:24:49.991046 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } attempt: 4
I20260923 20:24:49.992241 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.991713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49536)
0923 20:24:49.992016 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.992106 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260923 20:24:49.993286 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:49.992962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49546)
0923 20:24:49.993088 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:49.993173 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 20:24:49.993530 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45141: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45141: connect: Connection refused (error 111)
I20260923 20:24:49.995747 28357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:42891,127.14.127.253:45141,127.14.127.252:45899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab
--principal=kudu/127.14.127.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.253:45141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.127.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:49.997072 28357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:49.998404 28357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:50.010731 28362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:50.010731 28365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:50.010862 28363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:50.011507 28357 server_base.cc:1034] running on GCE node
I20260923 20:24:50.012235 28357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:50.013576 28357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:50.014802 28357 hybrid_clock.cc:648] HybridClock initialized: now 1790195090014753 us; error 42 us; skew 500 ppm
I20260923 20:24:50.015276 28357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:50.019814 28357 init.cc:377] Logged in from keytab as kudu/127.14.127.253@KRBTEST.COM (short username kudu)
I20260923 20:24:50.021683 28357 webserver.cc:460] Webserver started at http://127.14.127.253:32829/ using document root <none> and password file <none>
I20260923 20:24:50.022334 28357 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:50.022430 28357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:50.022753 28357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:24:50.025700 28357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/data/instance:
uuid: "f1c33ebaf0ea49559fa20aadbb531d95"
format_stamp: "Formatted at 2026-09-23 20:24:50 on dist-test-slave-wmd0"
server_key: "4ddbd0d793adba818539d89b72f3599d"
server_key_iv: "ab4ed269e94de23ad60ccc349dab1fc6"
server_key_version: "encryptionkey@0"
I20260923 20:24:50.026445 28357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/wal/instance:
uuid: "f1c33ebaf0ea49559fa20aadbb531d95"
format_stamp: "Formatted at 2026-09-23 20:24:50 on dist-test-slave-wmd0"
server_key: "4ddbd0d793adba818539d89b72f3599d"
server_key_iv: "ab4ed269e94de23ad60ccc349dab1fc6"
server_key_version: "encryptionkey@0"
I20260923 20:24:50.031459 28357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:24:50.034754 28372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:50.036022 28357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:24:50.036159 28357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/wal
uuid: "f1c33ebaf0ea49559fa20aadbb531d95"
format_stamp: "Formatted at 2026-09-23 20:24:50 on dist-test-slave-wmd0"
server_key: "4ddbd0d793adba818539d89b72f3599d"
server_key_iv: "ab4ed269e94de23ad60ccc349dab1fc6"
server_key_version: "encryptionkey@0"
I20260923 20:24:50.036327 28357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:50.068166 28357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:50.072577 28357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:50.072803 28357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:50.091317 28357 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.253:45141
I20260923 20:24:50.091400 28423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.253:45141 every 8 connection(s)
I20260923 20:24:50.092674 28357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
I20260923 20:24:50.095748 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 28357
I20260923 20:24:50.095947 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-1/wal/instance
I20260923 20:24:50.096351 28424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:50.096407 14847 external_mini_cluster.cc:1428] Setting key 67f1fafdb98790abaf13f2b158d973b7
I20260923 20:24:50.102743 28424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.127.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.252@KRBTEST.COM" created.
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:50.125609 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.105897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.14.127.254:42891)
0923 20:24:50.106136 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.106145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.107322 (+ 1177us) server_negotiation.cc:407] Connection header received
0923 20:24:50.109156 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.109164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.109277 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.109402 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.110862 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.112039 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.113545 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.114274 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.114428 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.119339 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.119344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.119347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.119379 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.122425 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.123229 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.123234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.123237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.123355 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.123874 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.123880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.123889 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.124168 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.124335 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.125054 (+ 719us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.125296 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:24:50.126536 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.106303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42891 (local address 127.0.0.1:52132)
0923 20:24:50.107146 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.107199 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:50.108894 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:50.109585 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:50.109592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:50.110076 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:50.110685 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.110704 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.112210 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.112214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.113394 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.113405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.114544 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.114554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.114858 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.115527 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:50.115578 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:50.119100 (+ 3522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:50.122580 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.122590 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.122627 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.123082 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.123476 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.123481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.123485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.123751 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.124310 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:50.124319 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:50.124667 (+ 348us) client_negotiation.cc:769] Sending connection context
0923 20:24:50.124939 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 20:24:50.125450 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":397,"threads_started":1}
I20260923 20:24:50.128293 28424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab.
Entry for principal kudu/127.14.127.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab.
I20260923 20:24:50.145951 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.252:45899
--webserver_interface=127.14.127.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab
--principal=kudu/127.14.127.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:42891,127.14.127.253:45141,127.14.127.252:45899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:24:50.147372 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.129102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49562)
0923 20:24:50.129277 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.129297 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:50.129402 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:50.130535 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:50.130544 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:50.131097 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:50.131325 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.131339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.132656 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.132664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.134229 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.134248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.135204 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.135210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.135453 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.136092 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:50.136119 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:50.140995 (+ 4876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:50.144766 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.144770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.144772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.145179 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.145557 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.145569 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.145572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.145705 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.146388 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:50.146392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:50.146531 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 20:24:50.146696 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 20:24:50.146980 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 20:24:50.147486 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.129406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.14.127.253:45141)
0923 20:24:50.130108 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.130114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.130138 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:24:50.130179 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.130185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.130358 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.130469 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.131491 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.132507 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.134418 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.135035 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.135144 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.141211 (+ 6067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.141222 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.141235 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.141264 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.144621 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.145309 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.145317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.145320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.145443 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.145815 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.145822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.145825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.146245 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.146467 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.146848 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.147103 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":551,"thread_start_us":409,"threads_started":1}
I20260923 20:24:50.149974 28424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:24:50.151144 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.150696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:37460)
0923 20:24:50.150848 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.150992 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 20:24:50.151408 28373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.152995 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.152658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:37472)
0923 20:24:50.152813 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.152898 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:24:50.153299 28424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.202644 28424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 1
I20260923 20:24:50.204035 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.203492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:37482)
0923 20:24:50.203695 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.203820 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:24:50.205229 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.204815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:37486)
0923 20:24:50.205064 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.205126 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260923 20:24:50.205461 28424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.273698 28424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 2
I20260923 20:24:50.274912 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.274416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:37500)
0923 20:24:50.274655 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.274762 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 20:24:50.275913 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.275592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:37508)
0923 20:24:50.275749 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.275809 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:24:50.276147 28424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.292874 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } attempt: 5
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195089, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:24:50.306375 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.293800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.14.127.253:45141)
0923 20:24:50.293982 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.293989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.294011 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:24:50.294229 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.294234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.294296 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.294401 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.295269 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.295945 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.297127 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.297592 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.297729 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.301090 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.301095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.301099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.301131 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.303860 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.304647 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.304651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.304653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.304766 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.305173 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.305175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.305177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.305448 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.305555 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.305985 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.306143 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:24:50.306581 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.293697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:49564)
0923 20:24:50.293914 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.293935 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:50.294078 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:50.294436 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:50.294440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:50.294912 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:50.295088 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.295097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.296108 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.296114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.296979 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.296991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.297765 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.297772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.297951 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.298590 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:50.298658 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:50.300880 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:50.304015 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.304020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.304024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.304526 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.304868 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.304873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.304877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.305050 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.305563 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:50.305568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:50.305737 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 20:24:50.305899 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 20:24:50.306245 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:24:50.307691 28350 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:24:50.308822 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.308372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48828)
0923 20:24:50.308561 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.308656 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 20:24:50.309926 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.309472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48834)
0923 20:24:50.309659 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.309760 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:24:50.310171 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.361443 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 1
I20260923 20:24:50.362689 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.362102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48840)
0923 20:24:50.362329 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.362453 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:24:50.363711 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.363335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48846)
0923 20:24:50.363483 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.363582 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 20:24:50.363921 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.367352 28424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 3
I20260923 20:24:50.368335 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.367907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48858)
0923 20:24:50.368101 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.368193 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 20:24:50.369393 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.368998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48862)
0923 20:24:50.369205 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.369287 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260923 20:24:50.369637 28424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.432199 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 2
I20260923 20:24:50.433523 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.432903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48878)
0923 20:24:50.433177 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.433300 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:24:50.434589 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.434284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48894)
0923 20:24:50.434419 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.434485 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:24:50.434866 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
W20260923 20:24:50.495504 28431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:50.495802 28431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:50.496008 28431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:50.503218 28431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:24:50.503320 28431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:50.503419 28431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:50.503506 28431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:24:50.503578 28431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:24:50.512883 28424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 4
I20260923 20:24:50.513676 28431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:42891,127.14.127.253:45141,127.14.127.252:45899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab
--principal=kudu/127.14.127.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.252:45899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.127.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:50.514065 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.513522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48908)
0923 20:24:50.513758 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.513852 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 20:24:50.515254 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.514870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48924)
0923 20:24:50.515035 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.515129 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 20:24:50.515460 28431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260923 20:24:50.515497 28424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.516745 28431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:50.528025 28436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:50.528025 28437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:50.528962 28439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:50.529986 28431 server_base.cc:1034] running on GCE node
I20260923 20:24:50.530857 28431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:50.532220 28431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:50.533437 28431 hybrid_clock.cc:648] HybridClock initialized: now 1790195090533391 us; error 36 us; skew 500 ppm
I20260923 20:24:50.533804 28431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:50.536990 28431 init.cc:377] Logged in from keytab as kudu/127.14.127.252@KRBTEST.COM (short username kudu)
I20260923 20:24:50.538400 28431 webserver.cc:460] Webserver started at http://127.14.127.252:39761/ using document root <none> and password file <none>
I20260923 20:24:50.539153 28431 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:50.539316 28431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:50.539683 28431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:24:50.541121 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 3
I20260923 20:24:50.542241 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.541738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48926)
0923 20:24:50.541990 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.542085 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 20:24:50.543356 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.542964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48928)
0923 20:24:50.543177 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.543253 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260923 20:24:50.543259 28431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/data/instance:
uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0"
format_stamp: "Formatted at 2026-09-23 20:24:50 on dist-test-slave-wmd0"
server_key: "504a7ce3b8f8b39753eca5250e38bfd4"
server_key_iv: "e9f582440d1374251ef1ddcd27ff1921"
server_key_version: "encryptionkey@0"
W20260923 20:24:50.543618 28350 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:45899: Network error: Client connection negotiation failed: client connection to 127.14.127.252:45899: connect: Connection refused (error 111)
I20260923 20:24:50.544153 28431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/wal/instance:
uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0"
format_stamp: "Formatted at 2026-09-23 20:24:50 on dist-test-slave-wmd0"
server_key: "504a7ce3b8f8b39753eca5250e38bfd4"
server_key_iv: "e9f582440d1374251ef1ddcd27ff1921"
server_key_version: "encryptionkey@0"
I20260923 20:24:50.549422 28431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:24:50.552851 28446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:50.554167 28431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:24:50.554356 28431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/wal
uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0"
format_stamp: "Formatted at 2026-09-23 20:24:50 on dist-test-slave-wmd0"
server_key: "504a7ce3b8f8b39753eca5250e38bfd4"
server_key_iv: "e9f582440d1374251ef1ddcd27ff1921"
server_key_version: "encryptionkey@0"
I20260923 20:24:50.554553 28431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:50.574429 28431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:50.578346 28431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:50.578693 28431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:50.597419 28431 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.252:45899
I20260923 20:24:50.597447 28497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.252:45899 every 8 connection(s)
I20260923 20:24:50.598695 28431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
I20260923 20:24:50.602181 28498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:50.608395 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 28431
I20260923 20:24:50.608587 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194882707199-14847-0/minicluster-data/master-2/wal/instance
I20260923 20:24:50.609019 14847 external_mini_cluster.cc:1428] Setting key 7a6056c992d299bd79c68f0f241295fe
I20260923 20:24:50.608556 28498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
I20260923 20:24:50.628644 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.611355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.14.127.254:42891)
0923 20:24:50.611577 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.611583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.612299 (+ 716us) server_negotiation.cc:407] Connection header received
0923 20:24:50.613884 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.613892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.613994 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.614094 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.615467 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.616171 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.617942 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.618641 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.618765 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.622496 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.622501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.622504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.622534 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.625480 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.626202 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.626209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.626215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.626354 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.626892 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.626899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.626902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.627164 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.627336 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.628054 (+ 718us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.628273 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 20:24:50.629766 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.611406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42891 (local address 127.0.0.1:49468)
0923 20:24:50.612152 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.612183 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:50.613653 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:50.614193 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:50.614199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:50.614910 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:50.615278 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.615294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.616328 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.616334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.617806 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.617817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.618850 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.618857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.619096 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.619853 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:50.619879 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:50.622302 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:50.625615 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.625628 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.625638 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.626083 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.626476 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.626480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.626484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.626780 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.627307 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:50.627316 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:50.627627 (+ 311us) client_negotiation.cc:769] Sending connection context
0923 20:24:50.627971 (+ 344us) client_negotiation.cc:240] Negotiation successful
0923 20:24:50.628431 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":3,"spinlock_wait_cycles":33664,"thread_start_us":410,"threads_started":1}
I20260923 20:24:50.629784 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.610901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.14.127.252:45899)
0923 20:24:50.611844 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.611860 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.611910 (+ 50us) server_negotiation.cc:407] Connection header received
0923 20:24:50.613642 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.613661 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.613875 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.614025 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.615544 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.617069 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.618733 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.619514 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.619643 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.623805 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.623813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.623824 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.623856 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.626529 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.627232 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.627246 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.627249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.627368 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.627754 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.627758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.627760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.628075 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.628386 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.628710 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.628950 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":686,"thread_start_us":443,"threads_started":1}
I20260923 20:24:50.631428 28498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:50.645085 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.632337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45141 (local address 127.0.0.1:55774)
0923 20:24:50.632568 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.632585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:50.632710 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:50.633134 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:50.633141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:50.633581 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:50.633791 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.633805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.634874 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.634878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.635829 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.635843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.636522 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.636528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.636713 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.637287 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:50.637304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:50.639493 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:50.642767 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.642773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.642777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.643137 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.643528 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.643532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.643536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.643689 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.644171 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:50.644176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:50.644321 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 20:24:50.644471 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 20:24:50.644744 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260923 20:24:50.645144 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.632221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.14.127.253:45141)
0923 20:24:50.632409 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.632414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.632689 (+ 275us) server_negotiation.cc:407] Connection header received
0923 20:24:50.632891 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.632899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.632985 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.633131 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.633978 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.634736 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.635971 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.636379 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.636471 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.639682 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.639687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.639690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.639722 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.642594 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.643258 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.643263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.643267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.643415 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.643796 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.643803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.643806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.644048 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.644206 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.644564 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.644773 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:24:50.646261 28498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:24:50.647248 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.632872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.14.127.254:42891)
0923 20:24:50.633025 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.633033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.633051 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:24:50.633234 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.633240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.633307 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.633386 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.634735 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.635509 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.637165 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.637572 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.637666 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.641919 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.641924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.641927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.641957 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.644555 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.645216 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.645223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.645226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.645468 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.645861 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.645867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.645870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.646127 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.646419 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.646716 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.646946 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:24:50.659992 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.647015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48958)
0923 20:24:50.647331 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.647349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:50.647436 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:50.647703 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:50.647706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:50.648057 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:50.648183 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.648190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.649456 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.649464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.650468 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.650479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.651238 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.651242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.651434 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.652013 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:50.652029 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:50.653842 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:50.657386 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.657394 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.657399 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.657774 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.658159 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.658164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.658167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.658357 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.659004 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:50.659008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:50.659146 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 20:24:50.659281 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 20:24:50.659586 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":17}
I20260923 20:24:50.660012 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.647272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.14.127.252:45899)
0923 20:24:50.647405 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.647410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.647425 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:24:50.647528 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.647532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.647585 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.647689 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.648574 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.649337 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.650599 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.651088 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.651226 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.654018 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.654022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.654025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.654050 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.656732 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.657890 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.657897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.657900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.658032 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.658455 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.658463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.658466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.658891 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.659039 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.659411 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.659726 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":45}
I20260923 20:24:50.664012 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.648856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.14.127.253:45141)
0923 20:24:50.649579 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.649584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.649599 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:24:50.649644 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.649649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.649714 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.649809 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.650976 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.651803 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.653055 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.653548 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.653679 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.658151 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.658156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.658159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.658188 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.661491 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.662130 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.662136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.662139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.662293 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.662646 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.662651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.662655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.663001 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.663269 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.663441 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.663661 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610}
I20260923 20:24:50.665907 28498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0: Bootstrap starting.
I20260923 20:24:50.669026 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:24:50.669513 28498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:50.670513 28498 log.cc:824] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:50.673310 28498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0: No bootstrap required, opened a new log
I20260923 20:24:50.685443 28498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
I20260923 20:24:50.685958 28498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:50.686066 28498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955eefcd07fa4bc6bf1ffacb161bd1c0, State: Initialized, Role: FOLLOWER
I20260923 20:24:50.686996 28498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
I20260923 20:24:50.688472 28511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.688969 28511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:24:50.690150 28498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:24:50.696908 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.675961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49486 (local address 127.14.127.254:42891)
0923 20:24:50.676279 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.676285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.677379 (+ 1094us) server_negotiation.cc:407] Connection header received
0923 20:24:50.677685 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.677693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.677797 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.677882 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.680978 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.682416 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.684754 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.685456 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.685569 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.689591 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.689597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.689602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.689646 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.693394 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.694370 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.694377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.694381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.694563 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.695058 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.695065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.695068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.695455 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.695715 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.696314 (+ 599us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.696544 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260923 20:24:50.702418 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.676068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.14.127.253:45141)
0923 20:24:50.676290 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.676300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.682735 (+ 6435us) server_negotiation.cc:407] Connection header received
0923 20:24:50.683165 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.683171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.683248 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.683313 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.685673 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.686786 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.689265 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.689913 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.690001 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.693722 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.693730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.693737 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.693784 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.697299 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.699848 (+ 2549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.699854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.699858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.700025 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.700549 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.700555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.700557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.700939 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.701175 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.701772 (+ 597us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.702020 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:24:50.707755 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.677377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.14.127.252:45899)
0923 20:24:50.677523 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.677531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.688199 (+ 10668us) server_negotiation.cc:407] Connection header received
0923 20:24:50.689233 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.689238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.689317 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.689392 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.691437 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.692826 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.695551 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.696576 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.696724 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.700773 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.700781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.700792 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.700824 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.704107 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.704976 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.704981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.704984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.705234 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.705726 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.705731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.705734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.706142 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.706407 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.706957 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.707220 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 20:24:50.710714 28522 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:24:50.710876 28522 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:24:50.707348 28431 master_runner.cc:186] Error getting master registration for 127.14.127.254:42891: OK, instance_id { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" instance_seqno: 1790195089581508 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:24:50.718948 28431 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:24:50.720883 28350 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 4
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195089, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
I20260923 20:24:50.735782 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.721449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48974)
0923 20:24:50.721666 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.721688 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:50.721811 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:50.722167 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:50.722173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:50.722714 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:50.722931 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.722943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.724035 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.724041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.725525 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.725540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.726463 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.726469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.726685 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.727289 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:50.727312 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:50.729716 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:50.732866 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.732872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.732876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.733241 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.733725 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.733730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.733733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.733920 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.734739 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:50.734747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:50.734931 (+ 184us) client_negotiation.cc:769] Sending connection context
0923 20:24:50.735109 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 20:24:50.735410 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:24:50.735947 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.721532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.14.127.252:45899)
0923 20:24:50.721681 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.721688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.721765 (+ 77us) server_negotiation.cc:407] Connection header received
0923 20:24:50.721935 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.721939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.722014 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.722086 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.723085 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.723892 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.725675 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.726315 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.726409 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.729905 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.729910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.729913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.729948 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.732721 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.733395 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.733402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.733410 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.733561 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.734055 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.734060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.734063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.734487 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.734664 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.735375 (+ 711us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.735589 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 20:24:50.741806 28350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: Bootstrap starting.
I20260923 20:24:50.745908 28350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:50.747301 28350 log.cc:824] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:50.750411 28350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: No bootstrap required, opened a new log
I20260923 20:24:50.761605 28350 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
I20260923 20:24:50.762041 28350 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:50.762178 28350 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c274fc195d4502ba9df761ac7376a5, State: Initialized, Role: FOLLOWER
I20260923 20:24:50.762867 28350 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
I20260923 20:24:50.763845 28525 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.764194 28525 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:24:50.765252 28350 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:24:50.774336 28511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:24:50.774675 28511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
W20260923 20:24:50.775549 28536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:24:50.775709 28536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:24:50.776312 28511 leader_election.cc:288] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9c274fc195d4502ba9df761ac7376a5 (127.14.127.254:42891), f1c33ebaf0ea49559fa20aadbb531d95 (127.14.127.253:45141)
I20260923 20:24:50.777036 28325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c274fc195d4502ba9df761ac7376a5" is_pre_election: true
I20260923 20:24:50.777737 28325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 955eefcd07fa4bc6bf1ffacb161bd1c0 in term 0.
I20260923 20:24:50.778407 28448 leader_election.cc:302] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 955eefcd07fa4bc6bf1ffacb161bd1c0, a9c274fc195d4502ba9df761ac7376a5; no voters:
I20260923 20:24:50.779879 28511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:24:50.780031 28511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:24:50.780128 28511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:50.783550 28511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
I20260923 20:24:50.784844 28325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c274fc195d4502ba9df761ac7376a5"
I20260923 20:24:50.785070 28325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:50.774180 28283 master_runner.cc:186] Error getting master registration for 127.14.127.253:45141: OK, instance_id { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" instance_seqno: 1790195090086027 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:24:50.787838 28283 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:24:50.788673 28325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 955eefcd07fa4bc6bf1ffacb161bd1c0 in term 1.
I20260923 20:24:50.789233 28448 leader_election.cc:302] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 955eefcd07fa4bc6bf1ffacb161bd1c0, a9c274fc195d4502ba9df761ac7376a5; no voters:
I20260923 20:24:50.790066 28538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:24:50.790877 28511 leader_election.cc:288] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [CANDIDATE]: Term 1 election: Requested vote from peers a9c274fc195d4502ba9df761ac7376a5 (127.14.127.254:42891), f1c33ebaf0ea49559fa20aadbb531d95 (127.14.127.253:45141)
I20260923 20:24:50.797456 28538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [term 1 LEADER]: Becoming Leader. State: Replica: 955eefcd07fa4bc6bf1ffacb161bd1c0, State: Running, Role: LEADER
I20260923 20:24:50.798094 28538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
I20260923 20:24:50.813287 28511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955eefcd07fa4bc6bf1ffacb161bd1c0. Latest consensus state: current_term: 1 leader_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.813616 28511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:50.814730 28424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } attempt: 5
W20260923 20:24:50.821069 28304 tablet.cc:2369] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:24:50.821382 28541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:24:50.823597 28541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 20:24:50 dist-test-slave-wmd0 krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195090, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
W20260923 20:24:50.830721 28452 tablet.cc:2369] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:24:50.833899 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.815315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:45899 (local address 127.0.0.1:48982)
0923 20:24:50.815669 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:50.815689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:50.815798 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:50.816641 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:50.816646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:50.817165 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:50.817368 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.817380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.819000 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.819006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.820447 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:50.820463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.821807 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.821813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:50.822040 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.822726 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:50.822750 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:50.825819 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:50.830530 (+ 4711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.830537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.830542 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.831084 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.831551 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:50.831557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:50.831561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:50.831827 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:50.832899 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:50.832907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:50.833056 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 20:24:50.833197 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 20:24:50.833491 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260923 20:24:50.835680 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:50.815989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.14.127.252:45899)
0923 20:24:50.816156 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:50.816162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:50.816180 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:24:50.816241 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:50.816246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:50.816324 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:50.816416 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:50.817610 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.818812 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.820630 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:50.821359 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:50.821485 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:50.826081 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:50.826092 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:50.826097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:50.826192 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:50.830381 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.831242 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.831249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.831252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.831405 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:50.832051 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:50.832058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:50.832062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:50.832692 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:50.834376 (+ 1684us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:50.834658 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 20:24:50.835050 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:24:50.839761 28325 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [term 1 FOLLOWER]: Refusing update from remote peer 955eefcd07fa4bc6bf1ffacb161bd1c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:24:50.840832 28511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:24:50.841328 28424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95: Bootstrap starting.
I20260923 20:24:50.845121 28424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:50.858494 28424 log.cc:824] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:50.858441 28525 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955eefcd07fa4bc6bf1ffacb161bd1c0. Latest consensus state: current_term: 1 leader_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.858793 28525 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:24:50.860392 28543 mvcc.cc:205] Tried to move back new op lower bound from 7332639092064399360 to 7332639091914670080. Current Snapshot: MvccSnapshot[applied={T|T < 7332639092064399360}]
I20260923 20:24:50.866871 28424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95: No bootstrap required, opened a new log
I20260923 20:24:50.870338 28542 mvcc.cc:205] Tried to move back new op lower bound from 7332639092064399360 to 7332639091914670080. Current Snapshot: MvccSnapshot[applied={T|T < 7332639092064399360}]
I20260923 20:24:50.886551 28539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.887004 28539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:50.887804 28541 catalog_manager.cc:1347] Generated new cluster ID: f7a2200d426e42cfbae9bd51834e3028
I20260923 20:24:50.887904 28541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:24:50.888598 28525 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.888865 28525 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:24:50.894170 28424 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
I20260923 20:24:50.895079 28424 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:50.895345 28424 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c33ebaf0ea49559fa20aadbb531d95, State: Initialized, Role: FOLLOWER
I20260923 20:24:50.896697 28424 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } }
I20260923 20:24:50.915220 28548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.915760 28548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:24:50.921248 28397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:50.919258 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1c33ebaf0ea49559fa20aadbb531d95"
I20260923 20:24:50.918773 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" is_pre_election: true
I20260923 20:24:50.924983 28397 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [term 1 FOLLOWER]: Refusing update from remote peer 955eefcd07fa4bc6bf1ffacb161bd1c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:24:50.926105 28539 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:24:50.927224 28424 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:24:50.935148 28541 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:24:50.937215 28541 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:24:50.979095 28538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.979455 28538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:50.986470 28541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0: Generated new TSK 0
I20260923 20:24:50.987658 28541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:24:50.990020 28548 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955eefcd07fa4bc6bf1ffacb161bd1c0. Latest consensus state: current_term: 1 leader_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.990376 28548 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:24:50.990029 28553 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c274fc195d4502ba9df761ac7376a5" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 42891 } } peers { permanent_uuid: "f1c33ebaf0ea49559fa20aadbb531d95" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45141 } } peers { permanent_uuid: "955eefcd07fa4bc6bf1ffacb161bd1c0" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 45899 } } }
I20260923 20:24:50.991009 28553 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:24:51.016553 28565 catalog_manager.cc:1259] Loaded cluster ID: f7a2200d426e42cfbae9bd51834e3028
I20260923 20:24:51.016710 28565 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95: loading cluster ID for follower catalog manager: success
I20260923 20:24:51.020535 28565 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95: acquiring CA information for follower catalog manager: success
I20260923 20:24:51.023195 28565 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1c33ebaf0ea49559fa20aadbb531d95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:24:51.777733 28536 catalog_manager.cc:1259] Loaded cluster ID: f7a2200d426e42cfbae9bd51834e3028
I20260923 20:24:51.777889 28536 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: loading cluster ID for follower catalog manager: success
I20260923 20:24:51.780405 28536 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: acquiring CA information for follower catalog manager: success
I20260923 20:24:51.782173 28536 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9c274fc195d4502ba9df761ac7376a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:24:52.163790 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 28283
I20260923 20:24:52.165169 28345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:52.280723 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 28283
I20260923 20:24:52.302557 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 28357
I20260923 20:24:52.303663 28419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:52.320755 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:52.319964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42891 (local address 127.0.0.1:49490)
0923 20:24:52.320464 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:52.320591 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":277,"threads_started":1}
W20260923 20:24:52.321098 28448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.254:42891: connect: Connection refused (error 111)
I20260923 20:24:52.322582 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:52.322147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:42891 (local address 127.0.0.1:49494)
0923 20:24:52.322316 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:52.322467 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:24:52.322887 28448 consensus_peers.cc:487] T 00000000000000000000000000000000 P 955eefcd07fa4bc6bf1ffacb161bd1c0 -> Peer a9c274fc195d4502ba9df761ac7376a5 (127.14.127.254:42891): Couldn't send request to peer a9c274fc195d4502ba9df761ac7376a5. Status: Network error: Client connection negotiation failed: client connection to 127.14.127.254:42891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 20:24:52.412890 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 28357
I20260923 20:24:52.433115 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 28431
I20260923 20:24:52.434171 28493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:52.542825 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 28431
2026-09-23T20:24:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4724 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:24:52 dist-test-slave-wmd0 krb5kdc[28577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:24:52 dist-test-slave-wmd0 krb5kdc[28577](info): set up 2 sockets
Sep 23 20:24:52 dist-test-slave-wmd0 krb5kdc[28577](info): commencing operation
krb5kdc: starting...
W20260923 20:24:53.639225 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:24:53 dist-test-slave-wmd0 krb5kdc[28577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:24:53.874310 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:39567
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:39567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:54.200488 28590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:54.200762 28590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:54.200855 28590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:54.207973 28590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:24:54.208071 28590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:54.208122 28590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:54.208180 28590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:24:54.208217 28590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:24:54.217844 28590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:39567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:39567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:54.219094 28590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:54.220510 28590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:54.232311 28596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:54.232270 28598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:54.232354 28595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:54.233239 28590 server_base.cc:1034] running on GCE node
I20260923 20:24:54.234020 28590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:54.235459 28590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:54.236687 28590 hybrid_clock.cc:648] HybridClock initialized: now 1790195094236646 us; error 54 us; skew 500 ppm
I20260923 20:24:54.237025 28590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:54 dist-test-slave-wmd0 krb5kdc[28577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:54.240051 28590 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:24:54.241466 28590 webserver.cc:460] Webserver started at http://127.14.127.254:33643/ using document root <none> and password file <none>
I20260923 20:24:54.242133 28590 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:54.242276 28590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:54.242590 28590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:24:54.245473 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "8e096ecb390e4d19b53615dbc3c02ce6"
format_stamp: "Formatted at 2026-09-23 20:24:54 on dist-test-slave-wmd0"
server_key: "f5d43b715645fdd9ef8549678a553702"
server_key_iv: "85d3129551a6dd2033d86cf9529cd498"
server_key_version: "encryptionkey@0"
I20260923 20:24:54.246225 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "8e096ecb390e4d19b53615dbc3c02ce6"
format_stamp: "Formatted at 2026-09-23 20:24:54 on dist-test-slave-wmd0"
server_key: "f5d43b715645fdd9ef8549678a553702"
server_key_iv: "85d3129551a6dd2033d86cf9529cd498"
server_key_version: "encryptionkey@0"
I20260923 20:24:54.251333 28590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 20:24:54.255071 28605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:54.256445 28590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:24:54.256629 28590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "8e096ecb390e4d19b53615dbc3c02ce6"
format_stamp: "Formatted at 2026-09-23 20:24:54 on dist-test-slave-wmd0"
server_key: "f5d43b715645fdd9ef8549678a553702"
server_key_iv: "85d3129551a6dd2033d86cf9529cd498"
server_key_version: "encryptionkey@0"
I20260923 20:24:54.256829 28590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:54.286098 28590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:54.290107 28590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:54.290418 28590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:54.309146 28590 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:39567
I20260923 20:24:54.309177 28656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:39567 every 8 connection(s)
I20260923 20:24:54.310530 28590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:24:54.314152 28657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:54.316491 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 28590
I20260923 20:24:54.316709 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:24:54.317366 14847 external_mini_cluster.cc:1428] Setting key dffe115b7c6fd7f3c5af634da07f1d28
I20260923 20:24:54.327138 28657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6: Bootstrap starting.
I20260923 20:24:54.330850 28657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:54.331830 28657 log.cc:824] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:24:54 dist-test-slave-wmd0 krb5kdc[28577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:54.334842 28657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6: No bootstrap required, opened a new log
I20260923 20:24:54.342636 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:54.319297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.14.127.254:39567)
0923 20:24:54.320103 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:54.320120 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:54.320170 (+ 50us) server_negotiation.cc:407] Connection header received
0923 20:24:54.322516 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:54.322542 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:54.323097 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:54.323540 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:54.324878 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:54.326850 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:54.328385 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:54.329494 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:54.329689 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:54.334039 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:54.334050 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:54.334069 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:54.334112 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:54.338124 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:54.338852 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:54.338862 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:54.338871 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:54.339089 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:54.339455 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:54.339460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:54.339464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:54.340148 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:54.340468 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:54.340815 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 20:24:54.341280 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":342,"threads_started":1}
I20260923 20:24:54.345607 28657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e096ecb390e4d19b53615dbc3c02ce6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39567 } }
I20260923 20:24:54.346086 28657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:54.346220 28657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e096ecb390e4d19b53615dbc3c02ce6, State: Initialized, Role: FOLLOWER
I20260923 20:24:54.346649 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:24:54.346918 28657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e096ecb390e4d19b53615dbc3c02ce6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39567 } }
I20260923 20:24:54.347179 28657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:24:54.347326 28657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:24:54.347499 28657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:54.350240 28657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e096ecb390e4d19b53615dbc3c02ce6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39567 } }
I20260923 20:24:54.350929 28657 leader_election.cc:302] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e096ecb390e4d19b53615dbc3c02ce6; no voters:
I20260923 20:24:54.351662 28657 leader_election.cc:288] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:24:54.351809 28662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:24:54.353322 28662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 8e096ecb390e4d19b53615dbc3c02ce6, State: Running, Role: LEADER
I20260923 20:24:54.353811 28662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e096ecb390e4d19b53615dbc3c02ce6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39567 } }
I20260923 20:24:54.354131 28657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:24:54.357125 28663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e096ecb390e4d19b53615dbc3c02ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e096ecb390e4d19b53615dbc3c02ce6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39567 } } }
I20260923 20:24:54.357596 28663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:54.357115 28664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e096ecb390e4d19b53615dbc3c02ce6. Latest consensus state: current_term: 1 leader_uuid: "8e096ecb390e4d19b53615dbc3c02ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e096ecb390e4d19b53615dbc3c02ce6" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 39567 } } }
I20260923 20:24:54.358122 28664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:54.361774 28670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:24:54.365873 28670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:24:54.373986 28670 catalog_manager.cc:1347] Generated new cluster ID: fd279f7a569c4296a928ba9f31dd7593
I20260923 20:24:54.374105 28670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:24:54.402308 28670 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:24:54.403708 28670 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:24:54.424244 28670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e096ecb390e4d19b53615dbc3c02ce6: Generated new TSK 0
I20260923 20:24:54.425133 28670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:24:54.459893 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 28590
I20260923 20:24:54.461033 28652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:54.559566 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 28590
2026-09-23T20:24:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2009 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:24:54 dist-test-slave-wmd0 krb5kdc[28685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:24:54 dist-test-slave-wmd0 krb5kdc[28685](info): set up 2 sockets
Sep 23 20:24:54 dist-test-slave-wmd0 krb5kdc[28685](info): commencing operation
krb5kdc: starting...
W20260923 20:24:55.663232 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:24:55 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:24:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:24:55.894539 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:43133
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:43133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:56.229086 28698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:56.229383 28698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:56.229475 28698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:56.236788 28698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:24:56.236902 28698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:56.236953 28698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:56.237010 28698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:24:56.237052 28698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:24:56.246878 28698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:43133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:43133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:56.248157 28698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:56.249408 28698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:56.260882 28706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:56.260941 28703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:56.260972 28704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:56.261808 28698 server_base.cc:1034] running on GCE node
I20260923 20:24:56.262506 28698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:56.263899 28698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:56.265103 28698 hybrid_clock.cc:648] HybridClock initialized: now 1790195096265063 us; error 25 us; skew 500 ppm
I20260923 20:24:56.265462 28698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:56 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195096, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:56.268651 28698 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:24:56.270114 28698 webserver.cc:460] Webserver started at http://127.14.127.254:38593/ using document root <none> and password file <none>
I20260923 20:24:56.270897 28698 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:56.271031 28698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:56.271354 28698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:24:56.274339 28698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "64cb309ea8ac47fd8f742dd614321b23"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "434267a9f3657bcfabb937e6ce5128d8"
server_key_iv: "6bae7042e5b59d535b97b0dd1fe6fb95"
server_key_version: "encryptionkey@0"
I20260923 20:24:56.275210 28698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "64cb309ea8ac47fd8f742dd614321b23"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "434267a9f3657bcfabb937e6ce5128d8"
server_key_iv: "6bae7042e5b59d535b97b0dd1fe6fb95"
server_key_version: "encryptionkey@0"
I20260923 20:24:56.280299 28698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:24:56.283695 28713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:56.285055 28698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:24:56.285240 28698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "64cb309ea8ac47fd8f742dd614321b23"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "434267a9f3657bcfabb937e6ce5128d8"
server_key_iv: "6bae7042e5b59d535b97b0dd1fe6fb95"
server_key_version: "encryptionkey@0"
I20260923 20:24:56.285435 28698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:56.315567 28698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:56.320052 28698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:56.320358 28698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:56.339910 28698 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:43133
I20260923 20:24:56.339932 28764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:43133 every 8 connection(s)
I20260923 20:24:56.341331 28698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:24:56.345002 28765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:56.346412 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 28698
I20260923 20:24:56.346702 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:24:56.347388 14847 external_mini_cluster.cc:1428] Setting key 69684d83d94f51e581931dcce47b02f2
I20260923 20:24:56.359975 28765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Bootstrap starting.
Sep 23 20:24:56 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:56.363446 28765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:56.364722 28765 log.cc:824] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:56.368629 28765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: No bootstrap required, opened a new log
I20260923 20:24:56.373291 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:56.349475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40356 (local address 127.14.127.254:43133)
0923 20:24:56.350297 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:56.350315 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:56.350382 (+ 67us) server_negotiation.cc:407] Connection header received
0923 20:24:56.352121 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:56.352152 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:56.352675 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:56.353121 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:56.354402 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:56.356506 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:56.358522 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:56.359580 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:56.359785 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:56.364264 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:56.364279 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:56.364299 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:56.364359 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:56.368467 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:56.369302 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:56.369312 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:56.369323 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:56.369559 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:56.369927 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:56.369934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:56.369940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:56.370709 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:56.371048 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:56.371363 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 20:24:56.371865 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":335,"threads_started":1}
I20260923 20:24:56.378389 28765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:24:56.378918 28765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:56.379007 28765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64cb309ea8ac47fd8f742dd614321b23, State: Initialized, Role: FOLLOWER
I20260923 20:24:56.379580 28765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:24:56.379860 28765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:24:56.380014 28765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:24:56.380206 28765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:56.382573 28765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:24:56.383155 28765 leader_election.cc:302] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64cb309ea8ac47fd8f742dd614321b23; no voters:
I20260923 20:24:56.383781 28765 leader_election.cc:288] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:24:56.383946 28770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:24:56.385233 28770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 1 LEADER]: Becoming Leader. State: Replica: 64cb309ea8ac47fd8f742dd614321b23, State: Running, Role: LEADER
I20260923 20:24:56.385771 28770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:24:56.386060 28765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:24:56.388549 28772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64cb309ea8ac47fd8f742dd614321b23. Latest consensus state: current_term: 1 leader_uuid: "64cb309ea8ac47fd8f742dd614321b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } } }
I20260923 20:24:56.388911 28772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:56.389569 28771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64cb309ea8ac47fd8f742dd614321b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } } }
I20260923 20:24:56.389910 28771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:56.391327 28778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:24:56.396893 28778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:24:56.405546 28778 catalog_manager.cc:1347] Generated new cluster ID: c2dff584996248d5b0712946680cf656
I20260923 20:24:56.405661 28778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:24:56.416535 28778 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:24:56.418267 28778 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:24:56.427366 28778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Generated new TSK 0
I20260923 20:24:56.428207 28778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:24:56.481432 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43133
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:56.809726 28791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:56.810055 28791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:56.810225 28791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:56.817458 28791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:56.817613 28791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:56.817801 28791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:24:56.827805 28791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:56.829087 28791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:56.830417 28791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:56.844340 28796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:56.844415 28799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:56.844305 28797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:56.845052 28791 server_base.cc:1034] running on GCE node
I20260923 20:24:56.845672 28791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:56.846787 28791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:56.848016 28791 hybrid_clock.cc:648] HybridClock initialized: now 1790195096847965 us; error 68 us; skew 500 ppm
I20260923 20:24:56.848356 28791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:56 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195096, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:56.851665 28791 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:24:56.853161 28791 webserver.cc:460] Webserver started at http://127.14.127.193:41537/ using document root <none> and password file <none>
I20260923 20:24:56.853874 28791 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:56.854027 28791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:56.854337 28791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:24:56.857353 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "a378201891c3416eb2da1cded37690c2"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "ba6fdd1e1d06335388c889615f68ce21"
server_key_iv: "df46d48a71cd37f4c8fc5a75b29ec4ac"
server_key_version: "encryptionkey@0"
I20260923 20:24:56.858146 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "a378201891c3416eb2da1cded37690c2"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "ba6fdd1e1d06335388c889615f68ce21"
server_key_iv: "df46d48a71cd37f4c8fc5a75b29ec4ac"
server_key_version: "encryptionkey@0"
I20260923 20:24:56.863802 28791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:24:56.867487 28806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:56.868896 28791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:24:56.869083 28791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "a378201891c3416eb2da1cded37690c2"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "ba6fdd1e1d06335388c889615f68ce21"
server_key_iv: "df46d48a71cd37f4c8fc5a75b29ec4ac"
server_key_version: "encryptionkey@0"
I20260923 20:24:56.869297 28791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:56.888283 28791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:56.892736 28791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:56.893035 28791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:56.894009 28791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:56.895465 28791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:56.895571 28791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:56.895738 28791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:56.895814 28791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:56.929230 28791 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:43327
I20260923 20:24:56.929275 28918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:43327 every 8 connection(s)
I20260923 20:24:56.930546 28791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:24:56.934408 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 28791
I20260923 20:24:56.934645 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:24:56.935384 14847 external_mini_cluster.cc:1428] Setting key 9045f734372c1979a2e2a34b7542e40b
Sep 23 20:24:56 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195096, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:56.953056 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:56.933174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:57167 (local address 127.14.127.254:43133)
0923 20:24:56.933634 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:56.933642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:56.934876 (+ 1234us) server_negotiation.cc:407] Connection header received
0923 20:24:56.936867 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:56.936872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:56.936986 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:56.937105 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:56.938664 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:56.939857 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:56.941376 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:56.942130 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:56.942282 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:56.945872 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:56.945882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:56.945890 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:56.945937 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:56.949642 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:56.950417 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:56.950425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:56.950431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:56.950586 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:56.951047 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:56.951055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:56.951061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:56.951460 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:56.951613 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:56.952314 (+ 701us) server_negotiation.cc:299] Negotiation successful
0923 20:24:56.952579 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260923 20:24:56.953768 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:56.933749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:57167)
0923 20:24:56.934682 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:56.934759 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:56.936597 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:56.937393 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:56.937400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:56.937905 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:56.938449 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:56.938466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:56.940007 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:56.940010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:56.941219 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:56.941230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:56.942355 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:56.942360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:56.942552 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:56.943324 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:56.943352 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:56.945645 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:56.949816 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:56.949823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:56.949836 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:56.950294 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:56.950752 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:56.950755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:56.950757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:56.950932 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:56.951590 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:56.951596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:56.951905 (+ 309us) client_negotiation.cc:769] Sending connection context
0923 20:24:56.952162 (+ 257us) client_negotiation.cc:240] Negotiation successful
0923 20:24:56.952551 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":330,"threads_started":1}
I20260923 20:24:56.955510 28919 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43133
I20260923 20:24:56.955816 28919 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:56.956552 28919 heartbeater.cc:511] Master 127.14.127.254:43133 requested a full tablet report, sending...
I20260923 20:24:56.959077 28730 ts_manager.cc:194] Registered new tserver with Master: a378201891c3416eb2da1cded37690c2 (127.14.127.193:43327)
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:24:56.960777 28730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:57167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:24:56.989943 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43133
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:57.312624 28925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:57.312908 28925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:57.313036 28925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:57.320518 28925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:57.320637 28925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:57.320828 28925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:24:57.331498 28925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:57.332855 28925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:57.334219 28925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:57.347843 28930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:57.347822 28931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:57.348459 28933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:57.348897 28925 server_base.cc:1034] running on GCE node
I20260923 20:24:57.349542 28925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:57.350591 28925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:57.351858 28925 hybrid_clock.cc:648] HybridClock initialized: now 1790195097351817 us; error 38 us; skew 500 ppm
I20260923 20:24:57.352222 28925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:57 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195097, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:57.355947 28925 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:24:57.359546 28925 webserver.cc:460] Webserver started at http://127.14.127.194:36135/ using document root <none> and password file <none>
I20260923 20:24:57.360248 28925 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:57.360402 28925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:57.360713 28925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:24:57.364876 28925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "e4b5fffa323347e68c3346ec5867f30a"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "aff44576c715fb8b81e4b3c8f82b2bbf"
server_key_iv: "c7506c1c2aaf3d3183890d59b2af2787"
server_key_version: "encryptionkey@0"
I20260923 20:24:57.365679 28925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "e4b5fffa323347e68c3346ec5867f30a"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "aff44576c715fb8b81e4b3c8f82b2bbf"
server_key_iv: "c7506c1c2aaf3d3183890d59b2af2787"
server_key_version: "encryptionkey@0"
I20260923 20:24:57.371235 28925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:24:57.374998 28940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:57.376367 28925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:24:57.376537 28925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "e4b5fffa323347e68c3346ec5867f30a"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "aff44576c715fb8b81e4b3c8f82b2bbf"
server_key_iv: "c7506c1c2aaf3d3183890d59b2af2787"
server_key_version: "encryptionkey@0"
I20260923 20:24:57.376742 28925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:57.400369 28925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:57.405017 28925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:57.405339 28925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:57.406315 28925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:57.407732 28925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:57.407836 28925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:57.407989 28925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:57.408062 28925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:57.441249 28925 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:44095
I20260923 20:24:57.441336 29052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:44095 every 8 connection(s)
I20260923 20:24:57.442549 28925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:24:57.442866 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 28925
I20260923 20:24:57.443055 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:24:57.443655 14847 external_mini_cluster.cc:1428] Setting key 85de6f5ced3fd1a1abce99e2d2010195
Sep 23 20:24:57 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195097, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:57.464581 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:57.445564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:55761 (local address 127.14.127.254:43133)
0923 20:24:57.445789 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:57.445795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:57.447223 (+ 1428us) server_negotiation.cc:407] Connection header received
0923 20:24:57.449025 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:57.449031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:57.449135 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:57.449247 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:57.451088 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.452085 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.453707 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.454157 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.454317 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:57.458719 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:57.458724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:57.458727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:57.458762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:57.461392 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:57.462160 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:57.462171 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:57.462173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:57.462305 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:57.462821 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:57.462825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:57.462828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:57.463104 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:57.463275 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:57.463981 (+ 706us) server_negotiation.cc:299] Negotiation successful
0923 20:24:57.464221 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:24:57.465593 29055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:57.446257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:55761)
0923 20:24:57.447076 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:57.447114 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:57.448766 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:57.449473 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:57.449482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:57.449978 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:57.450906 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:57.450927 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.452275 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.452288 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:57.453519 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:57.453531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.454430 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.454441 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:57.454838 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:57.455561 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:57.455594 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:57.458464 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:57.461552 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:57.461561 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:57.461580 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:57.462018 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:57.462423 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:57.462429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:57.462433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:57.462682 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:57.463248 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:57.463256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:57.463617 (+ 361us) client_negotiation.cc:769] Sending connection context
0923 20:24:57.463876 (+ 259us) client_negotiation.cc:240] Negotiation successful
0923 20:24:57.464446 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":362,"threads_started":1}
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
I20260923 20:24:57.467109 29053 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43133
I20260923 20:24:57.467442 29053 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:57.468264 29053 heartbeater.cc:511] Master 127.14.127.254:43133 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
I20260923 20:24:57.469817 28730 ts_manager.cc:194] Registered new tserver with Master: e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
I20260923 20:24:57.470700 28730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:55761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:24:57.495955 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43133
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:57.817410 29058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:57.817685 29058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:57.817778 29058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:57.825068 29058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:57.825227 29058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:57.825402 29058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:24:57.835035 29058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:57.836354 29058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:57.837783 29058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:57.850839 29064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:57.850899 29066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:57.850934 29063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:57.851727 29058 server_base.cc:1034] running on GCE node
I20260923 20:24:57.852327 29058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:57.853245 29058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:57.854511 29058 hybrid_clock.cc:648] HybridClock initialized: now 1790195097854464 us; error 52 us; skew 500 ppm
I20260923 20:24:57.854849 29058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:57 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195097, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:57.858116 29058 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:24:57.859627 29058 webserver.cc:460] Webserver started at http://127.14.127.195:41823/ using document root <none> and password file <none>
I20260923 20:24:57.860316 29058 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:57.860462 29058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:57.860761 29058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:24:57.863662 29058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "2a0b567a59e249ae8eac7c10e0001e46"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "3269b677c6e10d8ded3ae7e766db1371"
server_key_iv: "6e8d3ee9ca71b09ba53c89455db719b5"
server_key_version: "encryptionkey@0"
I20260923 20:24:57.864418 29058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "2a0b567a59e249ae8eac7c10e0001e46"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "3269b677c6e10d8ded3ae7e766db1371"
server_key_iv: "6e8d3ee9ca71b09ba53c89455db719b5"
server_key_version: "encryptionkey@0"
I20260923 20:24:57.869467 29058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:24:57.872933 29073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:57.874305 29058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:24:57.874485 29058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "2a0b567a59e249ae8eac7c10e0001e46"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "3269b677c6e10d8ded3ae7e766db1371"
server_key_iv: "6e8d3ee9ca71b09ba53c89455db719b5"
server_key_version: "encryptionkey@0"
I20260923 20:24:57.874773 29058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:57.895416 29058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:57.899776 29058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:57.900010 29058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:57.900905 29058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:57.902179 29058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:57.902261 29058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:57.902346 29058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:57.902390 29058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:57.935715 29058 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:42243
I20260923 20:24:57.935768 29185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:42243 every 8 connection(s)
I20260923 20:24:57.937100 29058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:24:57.938547 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 29058
I20260923 20:24:57.938781 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:24:57.939306 14847 external_mini_cluster.cc:1428] Setting key 18439c5deccb27a7c710cdcd4cf1395b
Sep 23 20:24:57 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195097, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:57.956614 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:57.939849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:55593 (local address 127.14.127.254:43133)
0923 20:24:57.940057 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:57.940063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:57.941501 (+ 1438us) server_negotiation.cc:407] Connection header received
0923 20:24:57.943399 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:57.943403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:57.943471 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:57.943555 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:57.944941 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.945679 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.947230 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.947617 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.947709 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:57.951039 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:57.951044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:57.951047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:57.951079 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:57.953786 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:57.954665 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:57.954671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:57.954674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:57.954837 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:57.955212 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:57.955215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:57.955217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:57.955476 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:57.955591 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:57.956215 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 20:24:57.956376 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:24:57.957561 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:57.940442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:55593)
0923 20:24:57.941344 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:57.941395 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:57.943135 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:57.943728 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:57.943736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:57.944235 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:57.944790 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:57.944810 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.945807 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.945810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:57.947111 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:57.947123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.947795 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.947799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:57.947983 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:57.948664 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:57.948684 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:57.950836 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:57.953930 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:57.953936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:57.953949 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:57.954516 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:57.954934 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:57.954937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:57.954940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:57.955112 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:57.955584 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:57.955589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:57.955893 (+ 304us) client_negotiation.cc:769] Sending connection context
0923 20:24:57.956113 (+ 220us) client_negotiation.cc:240] Negotiation successful
0923 20:24:57.956498 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":384,"threads_started":1}
I20260923 20:24:57.958961 29186 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43133
I20260923 20:24:57.959277 29186 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:57.959973 29186 heartbeater.cc:511] Master 127.14.127.254:43133 requested a full tablet report, sending...
I20260923 20:24:57.961387 28730 ts_manager.cc:194] Registered new tserver with Master: 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243)
I20260923 20:24:57.962121 28730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:55593
I20260923 20:24:57.963950 28919 heartbeater.cc:503] Master 127.14.127.254:43133 was elected leader, sending a full tablet report...
I20260923 20:24:57.966941 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:24:57.984158 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:57.971420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.14.127.254:43133)
0923 20:24:57.971613 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:57.971621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:57.972005 (+ 384us) server_negotiation.cc:407] Connection header received
0923 20:24:57.972175 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:57.972181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:57.972253 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:57.972361 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:57.973856 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.974574 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.975881 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:57.976324 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:57.976464 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:57.978935 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:57.978940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:57.978944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:57.978976 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:57.981465 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:57.982126 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:57.982133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:57.982137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:57.982282 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:57.982692 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:57.982697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:57.982700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:57.983029 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:57.983296 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:57.983612 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:24:57.983847 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:24:57.988274 28730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:24:57.995352 28730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:24:58.031078 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.006910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.14.127.193:43327)
0923 20:24:58.022825 (+ 15915us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.022833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.022864 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:24:58.022977 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.022985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.023233 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.024094 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:58.024403 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.026057 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.028659 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.029779 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.030012 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.030200 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.030542 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.030733 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15726,"thread_start_us":18556,"threads_started":1}
I20260923 20:24:58.031131 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.006720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:57198)
0923 20:24:58.007796 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:58.007838 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:58.008005 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:58.023514 (+ 15509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:58.023523 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:58.023562 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:58.024128 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.024148 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.026238 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.026245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.028451 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.028468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.030014 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.030020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.030304 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.030346 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:24:58.030498 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 20:24:58.030710 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":62,"spinlock_wait_cycles":9472,"thread_start_us":686,"threads_started":1}
I20260923 20:24:58.032750 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.007697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:42243 (local address 127.0.0.1:54142)
0923 20:24:58.022015 (+ 14318us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:58.022041 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:58.022231 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:58.023559 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:58.023567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:58.023606 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:58.025496 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.025508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.027474 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.027484 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.030001 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.030015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.031926 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.031932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.032091 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.032121 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:24:58.032223 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 20:24:58.032342 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14140,"mutex_wait_us":135,"thread_start_us":13958,"threads_started":1}
I20260923 20:24:58.032812 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.008422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.14.127.195:42243)
0923 20:24:58.020760 (+ 12338us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.020774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.022404 (+ 1630us) server_negotiation.cc:407] Connection header received
0923 20:24:58.022542 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.022554 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.022953 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.023194 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:58.025689 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.027367 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.030187 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.031660 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.031863 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.032029 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.032277 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.032444 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12088,"thread_start_us":12364,"threads_started":1}
I20260923 20:24:58.034062 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.008196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44095 (local address 127.0.0.1:34434)
0923 20:24:58.024139 (+ 15943us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:58.024158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:58.024283 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:58.025695 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:58.025699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:58.025733 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:58.026020 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.026027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.027428 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.027436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.030380 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.030394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.033293 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.033301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.033456 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.033486 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:24:58.033587 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:24:58.033703 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15798,"mutex_wait_us":129,"thread_start_us":15641,"threads_started":1}
I20260923 20:24:58.034116 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.008678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.14.127.194:44095)
0923 20:24:58.021460 (+ 12782us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.021472 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.024413 (+ 2941us) server_negotiation.cc:407] Connection header received
0923 20:24:58.024522 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.024532 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.024769 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.024945 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:58.026201 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.027282 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.030988 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.032415 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.032582 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.033324 (+ 742us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.033644 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.033807 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12599,"thread_start_us":12464,"threads_started":1}
I20260923 20:24:58.036130 28854 tablet_service.cc:1467] Processing CreateTablet for tablet 700b8c345db744e79bb3e0a39a2bb3f4 (DEFAULT_TABLE table=test-table [id=0c7472917728432889f3cb61d63f3be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:24:58.036388 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 700b8c345db744e79bb3e0a39a2bb3f4 (DEFAULT_TABLE table=test-table [id=0c7472917728432889f3cb61d63f3be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:24:58.037704 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700b8c345db744e79bb3e0a39a2bb3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:58.037707 28854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700b8c345db744e79bb3e0a39a2bb3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:58.037763 28988 tablet_service.cc:1467] Processing CreateTablet for tablet 700b8c345db744e79bb3e0a39a2bb3f4 (DEFAULT_TABLE table=test-table [id=0c7472917728432889f3cb61d63f3be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:24:58.038980 28988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 700b8c345db744e79bb3e0a39a2bb3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:24:58.049794 29204 tablet_bootstrap.cc:492] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Bootstrap starting.
I20260923 20:24:58.052184 29205 tablet_bootstrap.cc:492] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Bootstrap starting.
I20260923 20:24:58.053313 29204 tablet_bootstrap.cc:654] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:58.053889 29206 tablet_bootstrap.cc:492] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Bootstrap starting.
I20260923 20:24:58.055158 29204 log.cc:824] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:58.055912 29205 tablet_bootstrap.cc:654] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:58.057096 29206 tablet_bootstrap.cc:654] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Neither blocks nor log segments found. Creating new log.
I20260923 20:24:58.057108 29205 log.cc:824] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:58.058156 29206 log.cc:824] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:58.059885 29204 tablet_bootstrap.cc:492] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: No bootstrap required, opened a new log
I20260923 20:24:58.060221 29204 ts_tablet_manager.cc:1397] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 20:24:58.060691 29206 tablet_bootstrap.cc:492] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: No bootstrap required, opened a new log
I20260923 20:24:58.060948 29206 ts_tablet_manager.cc:1397] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 20:24:58.068401 29206 raft_consensus.cc:348] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.068809 29206 raft_consensus.cc:374] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:58.068849 29205 tablet_bootstrap.cc:492] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: No bootstrap required, opened a new log
I20260923 20:24:58.068913 29206 raft_consensus.cc:729] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b5fffa323347e68c3346ec5867f30a, State: Initialized, Role: FOLLOWER
I20260923 20:24:58.069173 29205 ts_tablet_manager.cc:1397] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.006s
I20260923 20:24:58.069600 29206 consensus_queue.cc:260] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.070739 29053 heartbeater.cc:503] Master 127.14.127.254:43133 was elected leader, sending a full tablet report...
I20260923 20:24:58.071429 29206 ts_tablet_manager.cc:1428] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260923 20:24:58.072745 29204 raft_consensus.cc:348] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.073170 29204 raft_consensus.cc:374] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:58.073315 29204 raft_consensus.cc:729] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0b567a59e249ae8eac7c10e0001e46, State: Initialized, Role: FOLLOWER
I20260923 20:24:58.073926 29204 consensus_queue.cc:260] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.075289 29186 heartbeater.cc:503] Master 127.14.127.254:43133 was elected leader, sending a full tablet report...
I20260923 20:24:58.078929 29204 ts_tablet_manager.cc:1428] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Time spent starting tablet: real 0.019s user 0.015s sys 0.001s
I20260923 20:24:58.079905 29205 raft_consensus.cc:348] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.080374 29205 raft_consensus.cc:374] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:24:58.080485 29205 raft_consensus.cc:729] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a378201891c3416eb2da1cded37690c2, State: Initialized, Role: FOLLOWER
I20260923 20:24:58.081051 29205 consensus_queue.cc:260] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.082368 29205 ts_tablet_manager.cc:1428] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260923 20:24:58.183022 28920 tablet.cc:2369] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:24:58.188666 29187 tablet.cc:2369] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:24:58.194744 29054 tablet.cc:2369] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:24:58.212847 29211 raft_consensus.cc:482] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:24:58.213184 29211 raft_consensus.cc:504] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.214985 29211 leader_election.cc:288] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a378201891c3416eb2da1cded37690c2 (127.14.127.193:43327), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
I20260923 20:24:58.221768 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.215360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.14.127.195:55189)
0923 20:24:58.215618 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:58.215667 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:58.215877 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:58.216528 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:58.216533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:58.216572 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:58.216793 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.216806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.217999 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.218005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.219821 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.219834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.220975 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.220982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.221179 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.221203 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:24:58.221311 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:24:58.221463 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 20:24:58.221768 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.215486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:55189 (local address 127.14.127.193:43327)
0923 20:24:58.215690 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.215698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.215749 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:24:58.216136 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.216144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.216239 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.216341 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:58.216943 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.217867 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.219999 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.220813 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.220997 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.221072 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.221346 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.221491 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:24:58.222117 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.216036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:36673 (local address 127.14.127.194:44095)
0923 20:24:58.216299 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.216305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.216661 (+ 356us) server_negotiation.cc:407] Connection header received
0923 20:24:58.216810 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.216813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.216871 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.216983 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:24:58.217581 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.218527 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.220433 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.221188 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.221371 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.221456 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.221750 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.221874 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:24:58.222167 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.215836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44095 (local address 127.14.127.195:36673)
0923 20:24:58.216563 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:58.216580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:58.216690 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:58.217015 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:58.217020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:58.217047 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:24:58.217421 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.217434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.218706 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.218711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.220277 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:58.220289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.221396 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.221402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:58.221593 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.221615 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:24:58.221730 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:24:58.221854 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":402,"threads_started":1}
I20260923 20:24:58.223199 29008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700b8c345db744e79bb3e0a39a2bb3f4" candidate_uuid: "2a0b567a59e249ae8eac7c10e0001e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b5fffa323347e68c3346ec5867f30a" is_pre_election: true
I20260923 20:24:58.223209 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700b8c345db744e79bb3e0a39a2bb3f4" candidate_uuid: "2a0b567a59e249ae8eac7c10e0001e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378201891c3416eb2da1cded37690c2" is_pre_election: true
I20260923 20:24:58.223696 29008 raft_consensus.cc:2454] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a0b567a59e249ae8eac7c10e0001e46 in term 0.
I20260923 20:24:58.223697 28874 raft_consensus.cc:2454] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a0b567a59e249ae8eac7c10e0001e46 in term 0.
I20260923 20:24:58.224298 29074 leader_election.cc:302] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a0b567a59e249ae8eac7c10e0001e46, e4b5fffa323347e68c3346ec5867f30a; no voters:
I20260923 20:24:58.224671 29211 raft_consensus.cc:2781] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:24:58.224839 29211 raft_consensus.cc:482] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:24:58.224946 29211 raft_consensus.cc:3037] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:58.227320 29211 raft_consensus.cc:504] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.227923 29211 leader_election.cc:288] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [CANDIDATE]: Term 1 election: Requested vote from peers a378201891c3416eb2da1cded37690c2 (127.14.127.193:43327), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
I20260923 20:24:58.228339 28874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700b8c345db744e79bb3e0a39a2bb3f4" candidate_uuid: "2a0b567a59e249ae8eac7c10e0001e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378201891c3416eb2da1cded37690c2"
I20260923 20:24:58.228403 29008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "700b8c345db744e79bb3e0a39a2bb3f4" candidate_uuid: "2a0b567a59e249ae8eac7c10e0001e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b5fffa323347e68c3346ec5867f30a"
I20260923 20:24:58.228539 28874 raft_consensus.cc:3037] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:58.228632 29008 raft_consensus.cc:3037] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:24:58.230803 28874 raft_consensus.cc:2454] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a0b567a59e249ae8eac7c10e0001e46 in term 1.
I20260923 20:24:58.230832 29008 raft_consensus.cc:2454] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a0b567a59e249ae8eac7c10e0001e46 in term 1.
I20260923 20:24:58.231225 29074 leader_election.cc:302] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a0b567a59e249ae8eac7c10e0001e46, e4b5fffa323347e68c3346ec5867f30a; no voters:
I20260923 20:24:58.231496 29211 raft_consensus.cc:2781] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:24:58.231928 29211 raft_consensus.cc:686] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 LEADER]: Becoming Leader. State: Replica: 2a0b567a59e249ae8eac7c10e0001e46, State: Running, Role: LEADER
I20260923 20:24:58.232393 29211 consensus_queue.cc:237] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } }
I20260923 20:24:58.236989 28728 catalog_manager.cc:5495] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "2a0b567a59e249ae8eac7c10e0001e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } health_report { overall_health: HEALTHY } } }
I20260923 20:24:58.276160 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.270779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.14.127.195:42243)
0923 20:24:58.271000 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.271004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.271019 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:24:58.271101 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.271105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.271185 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.271338 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:24:58.271733 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.272548 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.274036 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.274431 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.274564 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.275097 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:24:58.275583 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:24:58.275761 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.275822 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.275901 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:24:58.284260 29008 raft_consensus.cc:1261] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 1 FOLLOWER]: Refusing update from remote peer 2a0b567a59e249ae8eac7c10e0001e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:24:58.284344 28874 raft_consensus.cc:1261] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Refusing update from remote peer 2a0b567a59e249ae8eac7c10e0001e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:24:58.285259 29214 consensus_queue.cc:1035] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:24:58.285629 29211 consensus_queue.cc:1035] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:24:58.314755 28729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:24:58.315124 28729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:24:58.319407 28729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Sending DeleteTablet for 3 replicas of tablet 700b8c345db744e79bb3e0a39a2bb3f4
I20260923 20:24:58.320472 28854 tablet_service.cc:1514] Processing DeleteTablet for tablet 700b8c345db744e79bb3e0a39a2bb3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:24:58 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:57198
I20260923 20:24:58.320712 28988 tablet_service.cc:1514] Processing DeleteTablet for tablet 700b8c345db744e79bb3e0a39a2bb3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:24:58 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:34434
I20260923 20:24:58.320703 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet 700b8c345db744e79bb3e0a39a2bb3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:24:58 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:54142
I20260923 20:24:58.320852 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 28791
I20260923 20:24:58.335968 29228 tablet_replica.cc:330] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: stopping tablet replica
I20260923 20:24:58.336614 29229 tablet_replica.cc:330] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: stopping tablet replica
I20260923 20:24:58.337069 29229 raft_consensus.cc:2229] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:24:58.337759 29229 raft_consensus.cc:2258] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:24:58.338285 29227 tablet_replica.cc:330] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: stopping tablet replica
I20260923 20:24:58.338549 29228 raft_consensus.cc:2229] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:24:58.338801 29227 raft_consensus.cc:2229] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:24:58.339128 29228 raft_consensus.cc:2258] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:24:58.339344 29227 raft_consensus.cc:2258] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:24:58.341634 29228 ts_tablet_manager.cc:1905] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:24:58.341898 29227 ts_tablet_manager.cc:1905] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:24:58.344406 29229 ts_tablet_manager.cc:1905] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:24:58.350257 29228 ts_tablet_manager.cc:1918] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 23 20:24:58 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:24:58.357366 29229 ts_tablet_manager.cc:1918] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:24:58.357625 29228 log.cc:1196] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/700b8c345db744e79bb3e0a39a2bb3f4
I20260923 20:24:58.358208 29228 ts_tablet_manager.cc:1939] T 700b8c345db744e79bb3e0a39a2bb3f4 P e4b5fffa323347e68c3346ec5867f30a: Deleting consensus metadata
I20260923 20:24:58.360478 28714 catalog_manager.cc:4841] TS e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095): tablet 700b8c345db744e79bb3e0a39a2bb3f4 (table test-table [id=0c7472917728432889f3cb61d63f3be3]) successfully deleted
I20260923 20:24:58.362762 29229 log.cc:1196] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/700b8c345db744e79bb3e0a39a2bb3f4
I20260923 20:24:58.363504 29229 ts_tablet_manager.cc:1939] T 700b8c345db744e79bb3e0a39a2bb3f4 P 2a0b567a59e249ae8eac7c10e0001e46: Deleting consensus metadata
I20260923 20:24:58.363507 29227 ts_tablet_manager.cc:1918] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:24:58.363757 29227 log.cc:1196] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/700b8c345db744e79bb3e0a39a2bb3f4
I20260923 20:24:58.364334 29227 ts_tablet_manager.cc:1939] T 700b8c345db744e79bb3e0a39a2bb3f4 P a378201891c3416eb2da1cded37690c2: Deleting consensus metadata
I20260923 20:24:58.365154 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.322288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.14.127.193:43327)
0923 20:24:58.322509 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.322518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.337335 (+ 14817us) server_negotiation.cc:407] Connection header received
0923 20:24:58.337852 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.337861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.337967 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.338088 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:58.342743 (+ 4655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.343968 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.350756 (+ 6788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.351404 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.351515 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.357385 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:58.357401 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:58.357420 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:58.357471 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:58.361499 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:58.362361 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:58.362370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:58.362374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:58.362535 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:58.363042 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:58.363049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:58.363053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:58.363585 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:58.363928 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.364523 (+ 595us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.364770 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:24:58.365629 28715 catalog_manager.cc:4841] TS 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243): tablet 700b8c345db744e79bb3e0a39a2bb3f4 (table test-table [id=0c7472917728432889f3cb61d63f3be3]) successfully deleted
I20260923 20:24:58.366704 28914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:58.461191 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 28791
I20260923 20:24:58.461385 28714 catalog_manager.cc:4841] TS a378201891c3416eb2da1cded37690c2 (127.14.127.193:43327): tablet 700b8c345db744e79bb3e0a39a2bb3f4 (table test-table [id=0c7472917728432889f3cb61d63f3be3]) successfully deleted
I20260923 20:24:58.480332 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 28925
Sep 23 20:24:58 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:24:58.498257 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.481264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.14.127.194:44095)
0923 20:24:58.481478 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.481484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.481508 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:24:58.481686 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.481694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.481778 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.481897 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:58.483185 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.484349 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.485816 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.486460 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.486590 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.491520 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:58.491539 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:58.491559 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:58.491603 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:58.495430 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:58.496111 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:58.496121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:58.496125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:58.496292 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:58.496643 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:58.496648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:58.496652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:58.497132 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:58.497446 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.497635 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.497868 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:24:58.499210 29048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:58.588699 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 28925
I20260923 20:24:58.606175 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 29058
Sep 23 20:24:58 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:24:58.622581 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:58.607058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.14.127.195:42243)
0923 20:24:58.607288 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:58.607293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:58.607313 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:24:58.607471 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:58.607477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:58.607551 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:58.607666 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:58.608917 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.609746 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.611233 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:58.611687 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:58.611834 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:58.616440 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:58.616454 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:58.616469 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:58.616509 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:58.619693 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:58.620412 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:58.620427 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:58.620430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:58.620576 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:58.620957 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:58.620963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:58.620966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:58.621365 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:58.621676 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:58.621983 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 20:24:58.622227 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 20:24:58.623413 29181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:58.744721 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 29058
I20260923 20:24:58.764351 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 28698
I20260923 20:24:58.765530 28760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:24:58.876830 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 28698
I20260923 20:24:58.898087 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:43133
--webserver_interface=127.14.127.254
--webserver_port=38593
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:43133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:24:59.220826 29235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:59.221194 29235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:59.221360 29235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:59.228812 29235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:24:59.229064 29235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:59.229138 29235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:59.229192 29235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:24:59.229236 29235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:24:59.239135 29235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:43133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:43133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=38593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:59.240604 29235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:59.241845 29235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:59.266513 29241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:59.266458 29242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:59.266516 29244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:59.272207 29235 server_base.cc:1034] running on GCE node
I20260923 20:24:59.273116 29235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:59.274828 29235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:59.276067 29235 hybrid_clock.cc:648] HybridClock initialized: now 1790195099276021 us; error 55 us; skew 500 ppm
I20260923 20:24:59.276474 29235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:59 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195099, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:59.280047 29235 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:24:59.281824 29235 webserver.cc:460] Webserver started at http://127.14.127.254:38593/ using document root <none> and password file <none>
I20260923 20:24:59.282698 29235 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:59.282860 29235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:59.288836 29235 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260923 20:24:59.291885 29251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:59.293227 29235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:24:59.293414 29235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "64cb309ea8ac47fd8f742dd614321b23"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "434267a9f3657bcfabb937e6ce5128d8"
server_key_iv: "6bae7042e5b59d535b97b0dd1fe6fb95"
server_key_version: "encryptionkey@0"
I20260923 20:24:59.294487 29235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:59.310573 29235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:59.314735 29235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:59.315021 29235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:59.334401 29235 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:43133
I20260923 20:24:59.334422 29302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:43133 every 8 connection(s)
I20260923 20:24:59.336061 29235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:24:59.340662 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 29235
I20260923 20:24:59.341459 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:43327
--local_ip_for_outbound_sockets=127.14.127.193
--tserver_master_addrs=127.14.127.254:43133
--webserver_port=41537
--webserver_interface=127.14.127.193
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:24:59.343413 29303 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:24:59.346730 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Bootstrap starting.
I20260923 20:24:59.366259 29303 log.cc:824] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Log is configured to *not* fsync() on all Append() calls
I20260923 20:24:59.379688 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:24:59.380225 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Bootstrap complete.
I20260923 20:24:59.389441 29303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:24:59.390296 29303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64cb309ea8ac47fd8f742dd614321b23, State: Initialized, Role: FOLLOWER
I20260923 20:24:59.390940 29303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:24:59.391211 29303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:24:59.391352 29303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:24:59.391525 29303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:24:59.395535 29303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:24:59.396068 29303 leader_election.cc:302] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64cb309ea8ac47fd8f742dd614321b23; no voters:
I20260923 20:24:59.396728 29303 leader_election.cc:288] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:24:59.396871 29307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:24:59.398039 29307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 2 LEADER]: Becoming Leader. State: Replica: 64cb309ea8ac47fd8f742dd614321b23, State: Running, Role: LEADER
I20260923 20:24:59.398699 29307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:24:59.399042 29303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:24:59.401404 29308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "64cb309ea8ac47fd8f742dd614321b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } } }
I20260923 20:24:59.401847 29308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:59.403682 29312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:24:59.403549 29309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64cb309ea8ac47fd8f742dd614321b23. Latest consensus state: current_term: 2 leader_uuid: "64cb309ea8ac47fd8f742dd614321b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } } }
I20260923 20:24:59.403988 29309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: This master's current role is: LEADER
I20260923 20:24:59.411986 29312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:24:59.413561 29312 catalog_manager.cc:1259] Loaded cluster ID: c2dff584996248d5b0712946680cf656
I20260923 20:24:59.413756 29312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:24:59.417019 29312 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 20:24:59.435392 29312 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 20:24:59.454603 29312 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:24:59.456399 29312 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Loaded TSK: 0
I20260923 20:24:59.458086 29312 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:24:59.696171 29305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:24:59.696499 29305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:24:59.696672 29305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:24:59.704627 29305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:24:59.704788 29305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:24:59.704962 29305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:24:59.714845 29305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:43327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=41537
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:24:59.716106 29305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:24:59.717409 29305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:24:59.732115 29331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:59.732246 29330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:24:59.732448 29333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:24:59.736219 29305 server_base.cc:1034] running on GCE node
I20260923 20:24:59.736853 29305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:24:59.737805 29305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:24:59.739028 29305 hybrid_clock.cc:648] HybridClock initialized: now 1790195099738967 us; error 58 us; skew 500 ppm
I20260923 20:24:59.739344 29305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:24:59 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195099, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:24:59.748060 29305 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:24:59.749786 29305 webserver.cc:460] Webserver started at http://127.14.127.193:41537/ using document root <none> and password file <none>
I20260923 20:24:59.750523 29305 fs_manager.cc:359] Metadata directory not provided
I20260923 20:24:59.750715 29305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:24:59.756229 29305 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 20:24:59.759132 29340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:59.760553 29305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:24:59.760740 29305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "a378201891c3416eb2da1cded37690c2"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "ba6fdd1e1d06335388c889615f68ce21"
server_key_iv: "df46d48a71cd37f4c8fc5a75b29ec4ac"
server_key_version: "encryptionkey@0"
I20260923 20:24:59.761807 29305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:24:59.781883 29305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:24:59.786072 29305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:24:59.786365 29305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:24:59.787518 29305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:24:59.788915 29305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:24:59.789044 29305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:59.789186 29305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:24:59.789281 29305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:24:59.823140 29305 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:43327
I20260923 20:24:59.823230 29452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:43327 every 8 connection(s)
I20260923 20:24:59.824680 29305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:24:59.834486 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 29305
I20260923 20:24:59.835268 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:44095
--local_ip_for_outbound_sockets=127.14.127.194
--tserver_master_addrs=127.14.127.254:43133
--webserver_port=36135
--webserver_interface=127.14.127.194
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:24:59 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195099, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:24:59.848837 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:59.827635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:39557 (local address 127.14.127.254:43133)
0923 20:24:59.828210 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0923 20:24:59.828228 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:24:59.828868 (+ 640us) server_negotiation.cc:407] Connection header received
0923 20:24:59.831162 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:24:59.831175 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:24:59.831486 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:24:59.831849 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:24:59.833360 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:59.834687 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:59.836349 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:24:59.836969 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:24:59.837168 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:59.840953 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:24:59.840960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:24:59.840975 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:24:59.841011 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:24:59.844272 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:59.845007 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:59.845023 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:59.845031 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:59.845190 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:24:59.845664 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:24:59.845670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:24:59.845674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:24:59.846139 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:24:59.846340 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 20:24:59.847197 (+ 857us) server_negotiation.cc:299] Negotiation successful
0923 20:24:59.847565 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":264,"threads_started":1}
I20260923 20:24:59.848840 29456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:24:59.827883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:39557)
0923 20:24:59.828686 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0923 20:24:59.828733 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:24:59.830543 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:24:59.831989 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:24:59.832000 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:24:59.832608 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:24:59.833126 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:59.833143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:59.834863 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:59.834867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:59.836186 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:24:59.836199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:24:59.837211 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:24:59.837228 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:24:59.837485 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:24:59.838161 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:24:59.838187 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:24:59.840666 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:24:59.844416 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:59.844425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:59.844439 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:59.844872 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:59.845306 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:24:59.845311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:24:59.845315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:24:59.845545 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:24:59.846284 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:24:59.846292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:24:59.846685 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 20:24:59.846967 (+ 282us) client_negotiation.cc:240] Negotiation successful
0923 20:24:59.847528 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":394,"threads_started":1}
I20260923 20:24:59.851949 29453 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43133
I20260923 20:24:59.852252 29453 heartbeater.cc:464] Registering TS with master...
I20260923 20:24:59.852972 29453 heartbeater.cc:511] Master 127.14.127.254:43133 requested a full tablet report, sending...
I20260923 20:24:59.855115 29268 ts_manager.cc:194] Registered new tserver with Master: a378201891c3416eb2da1cded37690c2 (127.14.127.193:43327)
I20260923 20:24:59.856572 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:39557
W20260923 20:25:00.162807 29457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:00.163126 29457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:00.163303 29457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:00.170698 29457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:00.170861 29457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:00.171034 29457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:25:00.180456 29457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=36135
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:00.181715 29457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:00.183043 29457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:00.196094 29465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:00.196241 29462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:00.196183 29463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:00.196718 29457 server_base.cc:1034] running on GCE node
I20260923 20:25:00.197432 29457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:00.198521 29457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:00.199769 29457 hybrid_clock.cc:648] HybridClock initialized: now 1790195100199710 us; error 48 us; skew 500 ppm
I20260923 20:25:00.200103 29457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:00 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195100, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:00.203362 29457 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:25:00.204794 29457 webserver.cc:460] Webserver started at http://127.14.127.194:36135/ using document root <none> and password file <none>
I20260923 20:25:00.205472 29457 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:00.205650 29457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:00.211032 29457 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:25:00.213738 29472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:00.215147 29457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:25:00.215323 29457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "e4b5fffa323347e68c3346ec5867f30a"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "aff44576c715fb8b81e4b3c8f82b2bbf"
server_key_iv: "c7506c1c2aaf3d3183890d59b2af2787"
server_key_version: "encryptionkey@0"
I20260923 20:25:00.216298 29457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:00.231287 29457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:00.237059 29457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:00.237398 29457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:00.238799 29457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:00.240660 29457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:00.240783 29457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:00.240882 29457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:00.240975 29457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:00.280391 29457 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:44095
I20260923 20:25:00.280435 29584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:44095 every 8 connection(s)
I20260923 20:25:00.282609 29457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:25:00.289659 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 29457
I20260923 20:25:00.290511 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:42243
--local_ip_for_outbound_sockets=127.14.127.195
--tserver_master_addrs=127.14.127.254:43133
--webserver_port=41823
--webserver_interface=127.14.127.195
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:25:00 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195100, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:00.308374 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.285785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:51851 (local address 127.14.127.254:43133)
0923 20:25:00.285996 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.286003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.287712 (+ 1709us) server_negotiation.cc:407] Connection header received
0923 20:25:00.290458 (+ 2746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.290464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.290562 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.290697 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:00.292444 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.293155 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.295842 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.296523 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.296654 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.301002 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:00.301011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:00.301019 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:00.301065 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:00.304741 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:00.305496 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:00.305504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:00.305510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:00.305665 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:00.306095 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:00.306102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:00.306108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:00.306509 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:00.306722 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.307619 (+ 897us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.307900 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:25:00.308986 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.286547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:51851)
0923 20:25:00.287526 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:00.287582 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:00.290153 (+ 2571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:00.291143 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:00.291152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:00.291658 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:00.292260 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.292280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.293368 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.293377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.295654 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.295674 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.296794 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.296804 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.297146 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.298056 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:00.298080 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:00.300764 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:00.304903 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:00.304910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:00.304924 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:00.305351 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:00.305782 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:00.305785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:00.305787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:00.305964 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:00.306681 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:00.306689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:00.307212 (+ 523us) client_negotiation.cc:769] Sending connection context
0923 20:25:00.307445 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 20:25:00.307862 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":374,"threads_started":1}
I20260923 20:25:00.310536 29585 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43133
I20260923 20:25:00.310879 29585 heartbeater.cc:464] Registering TS with master...
I20260923 20:25:00.311626 29585 heartbeater.cc:511] Master 127.14.127.254:43133 requested a full tablet report, sending...
I20260923 20:25:00.313115 29268 ts_manager.cc:194] Registered new tserver with Master: e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
I20260923 20:25:00.313930 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:51851
W20260923 20:25:00.617772 29588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:00.618055 29588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:00.618224 29588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:00.625592 29588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:00.625730 29588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:00.625912 29588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:25:00.635787 29588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:42243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=41823
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:00.637140 29588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:00.638448 29588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:00.651669 29593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:00.651732 29594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:00.651703 29596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:00.652532 29588 server_base.cc:1034] running on GCE node
I20260923 20:25:00.653105 29588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:00.654014 29588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:00.655247 29588 hybrid_clock.cc:648] HybridClock initialized: now 1790195100655172 us; error 59 us; skew 500 ppm
I20260923 20:25:00.655562 29588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:00 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195100, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:00.659070 29588 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:25:00.660691 29588 webserver.cc:460] Webserver started at http://127.14.127.195:41823/ using document root <none> and password file <none>
I20260923 20:25:00.661432 29588 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:00.661607 29588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:00.667892 29588 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:25:00.671029 29603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:00.672463 29588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:25:00.672676 29588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "2a0b567a59e249ae8eac7c10e0001e46"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "3269b677c6e10d8ded3ae7e766db1371"
server_key_iv: "6e8d3ee9ca71b09ba53c89455db719b5"
server_key_version: "encryptionkey@0"
I20260923 20:25:00.673796 29588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:00.686863 29588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:00.691612 29588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:00.691903 29588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:00.692934 29588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:00.694319 29588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:00.694449 29588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:00.694595 29588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:00.694727 29588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260923 20:25:00.730756 29588 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:42243
I20260923 20:25:00.730791 29715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:42243 every 8 connection(s)
I20260923 20:25:00.732391 29588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:25:00.736845 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 29588
Sep 23 20:25:00 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195100, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:00.753573 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.737868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.14.127.254:43133)
0923 20:25:00.739055 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.739061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.739078 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:25:00.739130 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.739135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.739198 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.739339 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:00.740909 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.741738 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.743344 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.743862 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.743952 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.747657 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:00.747665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:00.747673 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:00.747728 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:00.750675 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:00.751464 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:00.751468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:00.751470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:00.751580 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:00.751998 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:00.752006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:00.752010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:00.752353 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:00.752622 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.752882 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.753125 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":1066,"spinlock_wait_cycles":3456,"thread_start_us":940,"threads_started":1}
I20260923 20:25:00.755517 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.735057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:37801 (local address 127.14.127.254:43133)
0923 20:25:00.735282 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.735289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.736613 (+ 1324us) server_negotiation.cc:407] Connection header received
0923 20:25:00.739072 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.739078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.739146 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.739234 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:00.740812 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.741616 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.743425 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.744018 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.744150 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.748507 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:00.748515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:00.748523 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:00.748578 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:00.751778 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:00.752697 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:00.752702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:00.752705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:00.752860 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:00.753365 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:00.753370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:00.753372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:00.753715 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:00.753861 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.754883 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.755119 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":76,"spinlock_wait_cycles":1792}
I20260923 20:25:00.756583 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.735657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:37801)
0923 20:25:00.736397 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:00.736456 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:00.738762 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:00.739538 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:00.739547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:00.740058 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:00.740626 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.740643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.741795 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.741803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.743261 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.743298 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.744241 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.744246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.744443 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.745164 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:00.745187 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:00.748283 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:00.751906 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:00.751912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:00.751942 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:00.752560 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:00.752979 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:00.752983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:00.752987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:00.753226 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:00.753928 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:00.753934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:00.754344 (+ 410us) client_negotiation.cc:769] Sending connection context
0923 20:25:00.754756 (+ 412us) client_negotiation.cc:240] Negotiation successful
0923 20:25:00.755298 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
I20260923 20:25:00.757884 29716 heartbeater.cc:347] Connected to a master server at 127.14.127.254:43133
I20260923 20:25:00.758209 29716 heartbeater.cc:464] Registering TS with master...
I20260923 20:25:00.759003 29716 heartbeater.cc:511] Master 127.14.127.254:43133 requested a full tablet report, sending...
I20260923 20:25:00.760378 29268 ts_manager.cc:194] Registered new tserver with Master: 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243)
I20260923 20:25:00.761123 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:37801
I20260923 20:25:00.768816 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:25:00.775655 29455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.769735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.14.127.254:43133)
0923 20:25:00.769903 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.769907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.770456 (+ 549us) server_negotiation.cc:407] Connection header received
0923 20:25:00.770597 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.770601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.770819 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.770959 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:25:00.771360 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.772162 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.773555 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.773994 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.774124 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.774596 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:25:00.775197 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:25:00.775317 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.775399 (+ 82us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.775461 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:25:00.776491 29268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:25:00.783880 29268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:25:00.811664 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.796852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.14.127.195:42243)
0923 20:25:00.798568 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.798575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.799658 (+ 1083us) server_negotiation.cc:407] Connection header received
0923 20:25:00.799872 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.799909 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.800196 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.800396 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:00.801733 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.804210 (+ 2477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.809048 (+ 4838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.810466 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.810691 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.810875 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.811141 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.811317 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1575,"thread_start_us":1426,"threads_started":1}
I20260923 20:25:00.811705 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.796112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:42243 (local address 127.0.0.1:34448)
0923 20:25:00.799368 (+ 3256us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:00.799417 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:00.799609 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:00.800798 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:00.800811 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:00.800857 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:00.801433 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.801450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.804456 (+ 3006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.804466 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.808874 (+ 4408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.808890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.810696 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.810703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.810898 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.810954 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 20:25:00.811089 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 20:25:00.811287 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3046,"mutex_wait_us":34,"thread_start_us":2822,"threads_started":1}
I20260923 20:25:00.812958 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.796138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.14.127.193:43327)
0923 20:25:00.797754 (+ 1616us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.797764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.800563 (+ 2799us) server_negotiation.cc:407] Connection header received
0923 20:25:00.800983 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.800994 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.801269 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.801454 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:00.802604 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.805921 (+ 3317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.809910 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.811383 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.811563 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.811737 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.812377 (+ 640us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.812586 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":1258,"threads_started":1}
I20260923 20:25:00.813822 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.796033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40358)
0923 20:25:00.800423 (+ 4390us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:00.800446 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:00.800768 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:00.801799 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:00.801806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:00.801841 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:00.802379 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.802394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.806158 (+ 3764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.806166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.809763 (+ 3597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.809777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.811899 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.811906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.812141 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.812186 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 20:25:00.813239 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0923 20:25:00.813406 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3053,"mutex_wait_us":431,"thread_start_us":2890,"threads_started":1}
I20260923 20:25:00.816062 29651 tablet_service.cc:1467] Processing CreateTablet for tablet 462c684450f04f42b96fa91fb60a61e5 (DEFAULT_TABLE table=test-table [id=53c82eafd2484fd5a0a4849f0d53c6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:00.817034 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.800069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44095 (local address 127.0.0.1:50914)
0923 20:25:00.808753 (+ 8684us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:00.808768 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:00.808920 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:00.809857 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:00.809862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:00.809891 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:00.810239 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.810250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.812221 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.812229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.814325 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.814337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.816172 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.816179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.816379 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.816418 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:25:00.816528 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:25:00.816701 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8559,"thread_start_us":8371,"threads_started":1}
I20260923 20:25:00.817054 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.800191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.14.127.194:44095)
0923 20:25:00.807780 (+ 7589us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.807787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.808878 (+ 1091us) server_negotiation.cc:407] Connection header received
0923 20:25:00.809147 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.809156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.809414 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.809605 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:00.810429 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.812043 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.814484 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.815932 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.816108 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.816264 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.816583 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.816764 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7422,"thread_start_us":7260,"threads_started":1}
I20260923 20:25:00.817612 29651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462c684450f04f42b96fa91fb60a61e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:00.818368 29388 tablet_service.cc:1467] Processing CreateTablet for tablet 462c684450f04f42b96fa91fb60a61e5 (DEFAULT_TABLE table=test-table [id=53c82eafd2484fd5a0a4849f0d53c6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:00.819818 29388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462c684450f04f42b96fa91fb60a61e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:00.820842 29520 tablet_service.cc:1467] Processing CreateTablet for tablet 462c684450f04f42b96fa91fb60a61e5 (DEFAULT_TABLE table=test-table [id=53c82eafd2484fd5a0a4849f0d53c6fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:00.822172 29520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462c684450f04f42b96fa91fb60a61e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:00.838507 29732 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Bootstrap starting.
I20260923 20:25:00.839684 29734 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: Bootstrap starting.
I20260923 20:25:00.842259 29732 tablet_bootstrap.cc:654] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:00.842274 29733 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Bootstrap starting.
I20260923 20:25:00.843325 29734 tablet_bootstrap.cc:654] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:00.844659 29734 log.cc:824] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:00.844659 29732 log.cc:824] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:00.845780 29733 tablet_bootstrap.cc:654] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:00.847105 29733 log.cc:824] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:00.849742 29733 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: No bootstrap required, opened a new log
I20260923 20:25:00.850023 29733 ts_tablet_manager.cc:1397] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 20:25:00.852871 29734 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: No bootstrap required, opened a new log
I20260923 20:25:00.853134 29734 ts_tablet_manager.cc:1397] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260923 20:25:00.853540 29732 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: No bootstrap required, opened a new log
I20260923 20:25:00.853801 29732 ts_tablet_manager.cc:1397] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260923 20:25:00.859472 29453 heartbeater.cc:503] Master 127.14.127.254:43133 was elected leader, sending a full tablet report...
I20260923 20:25:00.861640 29734 raft_consensus.cc:348] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.861645 29733 raft_consensus.cc:348] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.862105 29733 raft_consensus.cc:374] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:00.862214 29733 raft_consensus.cc:729] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a378201891c3416eb2da1cded37690c2, State: Initialized, Role: FOLLOWER
I20260923 20:25:00.862273 29734 raft_consensus.cc:374] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:00.862502 29734 raft_consensus.cc:729] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b5fffa323347e68c3346ec5867f30a, State: Initialized, Role: FOLLOWER
I20260923 20:25:00.863276 29734 consensus_queue.cc:260] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.862994 29733 consensus_queue.cc:260] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.865389 29733 ts_tablet_manager.cc:1428] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260923 20:25:00.865389 29732 raft_consensus.cc:348] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.865851 29732 raft_consensus.cc:374] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:00.865988 29732 raft_consensus.cc:729] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0b567a59e249ae8eac7c10e0001e46, State: Initialized, Role: FOLLOWER
I20260923 20:25:00.866653 29732 consensus_queue.cc:260] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.868615 29732 ts_tablet_manager.cc:1428] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260923 20:25:00.869323 29716 heartbeater.cc:503] Master 127.14.127.254:43133 was elected leader, sending a full tablet report...
I20260923 20:25:00.870759 29585 heartbeater.cc:503] Master 127.14.127.254:43133 was elected leader, sending a full tablet report...
I20260923 20:25:00.871950 29734 ts_tablet_manager.cc:1428] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: Time spent starting tablet: real 0.019s user 0.008s sys 0.010s
I20260923 20:25:00.953153 29740 raft_consensus.cc:482] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:25:00.953486 29740 raft_consensus.cc:504] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.955189 29740 leader_election.cc:288] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a378201891c3416eb2da1cded37690c2 (127.14.127.193:43327), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
I20260923 20:25:00.961947 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.955672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.14.127.195:41403)
0923 20:25:00.955910 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:00.955931 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:00.956119 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:00.956602 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:00.956609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:00.956645 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:00.956893 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.956905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.958214 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.958219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.960083 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.960098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.961430 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.961434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.961558 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.961582 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:25:00.961652 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 20:25:00.961748 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 20:25:00.962347 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.955687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:41403 (local address 127.14.127.193:43327)
0923 20:25:00.955913 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.955920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.956020 (+ 100us) server_negotiation.cc:407] Connection header received
0923 20:25:00.956305 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.956314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.956408 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.956527 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:00.957066 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.958081 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.960264 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.961277 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.961430 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.961507 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.961708 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.961844 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:25:00.962910 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.956256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:53863 (local address 127.14.127.194:44095)
0923 20:25:00.956472 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.956481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.957018 (+ 537us) server_negotiation.cc:407] Connection header received
0923 20:25:00.957202 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.957207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.957282 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.957378 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:00.957817 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.958665 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.960745 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.961921 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.962069 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.962179 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.962498 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.962652 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:25:00.962920 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.956082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44095 (local address 127.14.127.195:53863)
0923 20:25:00.956905 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:00.956922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:00.957036 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:00.957397 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:00.957400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:00.957418 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:00.957673 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.957680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.958826 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.958832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.960590 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:00.960599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.962136 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.962142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:00.962337 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.962359 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:25:00.962455 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 20:25:00.962569 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":419,"threads_started":1}
I20260923 20:25:00.963450 29408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462c684450f04f42b96fa91fb60a61e5" candidate_uuid: "2a0b567a59e249ae8eac7c10e0001e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378201891c3416eb2da1cded37690c2" is_pre_election: true
I20260923 20:25:00.963907 29540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462c684450f04f42b96fa91fb60a61e5" candidate_uuid: "2a0b567a59e249ae8eac7c10e0001e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b5fffa323347e68c3346ec5867f30a" is_pre_election: true
I20260923 20:25:00.963986 29408 raft_consensus.cc:2454] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a0b567a59e249ae8eac7c10e0001e46 in term 0.
I20260923 20:25:00.964349 29540 raft_consensus.cc:2454] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a0b567a59e249ae8eac7c10e0001e46 in term 0.
I20260923 20:25:00.964707 29604 leader_election.cc:302] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a0b567a59e249ae8eac7c10e0001e46, a378201891c3416eb2da1cded37690c2; no voters:
I20260923 20:25:00.965085 29740 raft_consensus.cc:2781] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:25:00.965235 29740 raft_consensus.cc:482] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:25:00.965358 29740 raft_consensus.cc:3037] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:00.967689 29740 raft_consensus.cc:504] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.968290 29740 leader_election.cc:288] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [CANDIDATE]: Term 1 election: Requested vote from peers a378201891c3416eb2da1cded37690c2 (127.14.127.193:43327), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
I20260923 20:25:00.968771 29540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462c684450f04f42b96fa91fb60a61e5" candidate_uuid: "2a0b567a59e249ae8eac7c10e0001e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b5fffa323347e68c3346ec5867f30a"
I20260923 20:25:00.968748 29408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "462c684450f04f42b96fa91fb60a61e5" candidate_uuid: "2a0b567a59e249ae8eac7c10e0001e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a378201891c3416eb2da1cded37690c2"
I20260923 20:25:00.968972 29540 raft_consensus.cc:3037] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:00.968989 29408 raft_consensus.cc:3037] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:00.971294 29540 raft_consensus.cc:2454] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a0b567a59e249ae8eac7c10e0001e46 in term 1.
I20260923 20:25:00.971364 29408 raft_consensus.cc:2454] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a0b567a59e249ae8eac7c10e0001e46 in term 1.
I20260923 20:25:00.971732 29604 leader_election.cc:302] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a0b567a59e249ae8eac7c10e0001e46, e4b5fffa323347e68c3346ec5867f30a; no voters:
I20260923 20:25:00.972023 29740 raft_consensus.cc:2781] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:25:00.972448 29740 raft_consensus.cc:686] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 LEADER]: Becoming Leader. State: Replica: 2a0b567a59e249ae8eac7c10e0001e46, State: Running, Role: LEADER
I20260923 20:25:00.972910 29740 consensus_queue.cc:237] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:00.977699 29268 catalog_manager.cc:5495] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "2a0b567a59e249ae8eac7c10e0001e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } health_report { overall_health: UNKNOWN } } }
W20260923 20:25:00.984020 29717 tablet.cc:2369] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:25:00.997990 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:00.992837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.14.127.195:42243)
0923 20:25:00.993016 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:00.993020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:00.993073 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:25:00.993244 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:00.993248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:00.993340 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:00.993458 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:25:00.993766 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.994454 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.995981 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:00.996368 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:00.996458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:00.996889 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:25:00.997472 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:25:00.997578 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:00.997687 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 20:25:00.997751 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:25:01.006578 29540 raft_consensus.cc:1261] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 1 FOLLOWER]: Refusing update from remote peer 2a0b567a59e249ae8eac7c10e0001e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:25:01.006578 29408 raft_consensus.cc:1261] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Refusing update from remote peer 2a0b567a59e249ae8eac7c10e0001e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:25:01.007622 29740 consensus_queue.cc:1035] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:25:01.007942 29742 consensus_queue.cc:1035] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:25:01.034665 29267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:25:01.035002 29267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:25:01.038776 29267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Sending DeleteTablet for 3 replicas of tablet 462c684450f04f42b96fa91fb60a61e5
I20260923 20:25:01.039953 29651 tablet_service.cc:1514] Processing DeleteTablet for tablet 462c684450f04f42b96fa91fb60a61e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:01 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:34448
I20260923 20:25:01.039985 29388 tablet_service.cc:1514] Processing DeleteTablet for tablet 462c684450f04f42b96fa91fb60a61e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:01 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:40358
I20260923 20:25:01.040129 29520 tablet_service.cc:1514] Processing DeleteTablet for tablet 462c684450f04f42b96fa91fb60a61e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:01 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:50914
I20260923 20:25:01.041051 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 29305
I20260923 20:25:01.052945 29754 tablet_replica.cc:330] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: stopping tablet replica
I20260923 20:25:01.053534 29754 raft_consensus.cc:2229] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:25:01.054207 29754 raft_consensus.cc:2258] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:01.055155 29755 tablet_replica.cc:330] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: stopping tablet replica
I20260923 20:25:01.055578 29756 tablet_replica.cc:330] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: stopping tablet replica
I20260923 20:25:01.055598 29755 raft_consensus.cc:2229] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:25:01.055995 29756 raft_consensus.cc:2229] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:25:01.056348 29755 raft_consensus.cc:2258] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:01.056460 29756 raft_consensus.cc:2258] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:01.058492 29755 ts_tablet_manager.cc:1905] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:25:01.058481 29756 ts_tablet_manager.cc:1905] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:25:01.063167 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.041845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.14.127.193:43327)
0923 20:25:01.042449 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:01.042455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:01.042472 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:25:01.042525 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:01.042530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:01.042662 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:01.042985 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:01.044941 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:01.046129 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:01.048502 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:01.049167 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:01.049290 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:01.053702 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:01.053718 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:01.053736 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:01.053786 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:01.058690 (+ 4904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:01.059718 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:01.059726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:01.059730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:01.059992 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:01.060475 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:01.060480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:01.060484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:01.060966 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:01.061258 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:01.062491 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0923 20:25:01.062743 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479}
I20260923 20:25:01.064147 29448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:01.066671 29756 ts_tablet_manager.cc:1918] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:25:01.066735 29755 ts_tablet_manager.cc:1918] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:25:01.066962 29756 log.cc:1196] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/462c684450f04f42b96fa91fb60a61e5
I20260923 20:25:01.066963 29755 log.cc:1196] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/462c684450f04f42b96fa91fb60a61e5
I20260923 20:25:01.067682 29756 ts_tablet_manager.cc:1939] T 462c684450f04f42b96fa91fb60a61e5 P e4b5fffa323347e68c3346ec5867f30a: Deleting consensus metadata
I20260923 20:25:01.067720 29755 ts_tablet_manager.cc:1939] T 462c684450f04f42b96fa91fb60a61e5 P 2a0b567a59e249ae8eac7c10e0001e46: Deleting consensus metadata
I20260923 20:25:01.069681 29252 catalog_manager.cc:4841] TS e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095): tablet 462c684450f04f42b96fa91fb60a61e5 (table test-table [id=53c82eafd2484fd5a0a4849f0d53c6fa]) successfully deleted
I20260923 20:25:01.069904 29253 catalog_manager.cc:4841] TS 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243): tablet 462c684450f04f42b96fa91fb60a61e5 (table test-table [id=53c82eafd2484fd5a0a4849f0d53c6fa]) successfully deleted
I20260923 20:25:01.165027 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 29305
I20260923 20:25:01.189438 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 29457
W20260923 20:25:01.189529 29252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.127.193:43327 (error 108)
I20260923 20:25:01.193181 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.192750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40382)
0923 20:25:01.192922 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:01.193042 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 20:25:01.193658 29252 catalog_manager.cc:4584] TS a378201891c3416eb2da1cded37690c2 (127.14.127.193:43327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 462c684450f04f42b96fa91fb60a61e5: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
I20260923 20:25:01.206348 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.190403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.14.127.194:44095)
0923 20:25:01.191173 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:01.191182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:01.191206 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:25:01.191286 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:01.191295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:01.191395 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:01.191518 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:01.193471 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:01.194412 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:01.196446 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:01.196842 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:01.196936 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:01.200437 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:01.200451 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:01.200466 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:01.200503 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:01.203512 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:01.204379 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:01.204386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:01.204388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:01.204513 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:01.204900 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:01.204903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:01.204905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:01.205250 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:01.205465 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:01.205813 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 20:25:01.205969 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589}
I20260923 20:25:01.207166 29580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:01.247473 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.246983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40396)
0923 20:25:01.247178 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:01.247282 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:25:01.248734 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.248310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40406)
0923 20:25:01.248481 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:01.248572 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:25:01.297356 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 29457
I20260923 20:25:01.310235 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.309566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40414)
0923 20:25:01.309873 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:01.309995 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260923 20:25:01.311568 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.311123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40420)
0923 20:25:01.311308 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:01.311411 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:25:01.315752 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 29588
I20260923 20:25:01.330539 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.316540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.14.127.195:42243)
0923 20:25:01.316730 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:01.316737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:01.316758 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:25:01.316962 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:01.316967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:01.317056 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:01.317173 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:01.318485 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:01.319390 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:01.320851 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:01.321324 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:01.321461 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:01.324307 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:01.324328 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:01.324343 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:01.324384 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:01.327599 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:01.328338 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:01.328346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:01.328348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:01.328500 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:01.328900 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:01.328905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:01.328910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:01.329291 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:01.329625 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:01.329923 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 20:25:01.330155 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:25:01.331504 29711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:01.392870 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.392278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40424)
0923 20:25:01.392523 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:01.392677 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 20:25:01.393955 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.393526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40432)
0923 20:25:01.393737 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:01.393822 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260923 20:25:01.456200 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 29588
I20260923 20:25:01.476403 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 29235
I20260923 20:25:01.477648 29298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:01.591821 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:01.591106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:43327 (local address 127.0.0.1:40448)
0923 20:25:01.591466 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:01.591620 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260923 20:25:01.591869 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 29235
I20260923 20:25:01.610503 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:43133
--webserver_interface=127.14.127.254
--webserver_port=38593
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:43133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:25:01.942947 29762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:01.943261 29762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:01.943354 29762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:01.950736 29762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:25:01.950850 29762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:01.950896 29762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:01.950953 29762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:25:01.950997 29762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:25:01.960775 29762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:43133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:43133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=38593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:01.962112 29762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:01.963393 29762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:01.975574 29770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:01.975585 29768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:01.975636 29767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:01.976531 29762 server_base.cc:1034] running on GCE node
I20260923 20:25:01.977401 29762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:01.978937 29762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:01.980197 29762 hybrid_clock.cc:648] HybridClock initialized: now 1790195101980111 us; error 48 us; skew 500 ppm
I20260923 20:25:01.980554 29762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:01 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195101, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:01.983788 29762 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:25:01.985349 29762 webserver.cc:460] Webserver started at http://127.14.127.254:38593/ using document root <none> and password file <none>
I20260923 20:25:01.986039 29762 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:01.986195 29762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:01.991787 29762 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:25:01.994762 29777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:01.996129 29762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:25:01.996351 29762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "64cb309ea8ac47fd8f742dd614321b23"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "434267a9f3657bcfabb937e6ce5128d8"
server_key_iv: "6bae7042e5b59d535b97b0dd1fe6fb95"
server_key_version: "encryptionkey@0"
I20260923 20:25:01.997367 29762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:02.022588 29762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:02.026876 29762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:02.027176 29762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:02.045343 29762 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:43133
I20260923 20:25:02.045322 29828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:43133 every 8 connection(s)
I20260923 20:25:02.046970 29762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:25:02.053469 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 29762
I20260923 20:25:02.054163 29829 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:25:02.054373 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:43327
--local_ip_for_outbound_sockets=127.14.127.193
--tserver_master_addrs=127.14.127.254:43133
--webserver_port=41537
--webserver_interface=127.14.127.193
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:25:02.057456 29829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Bootstrap starting.
I20260923 20:25:02.078317 29829 log.cc:824] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:02.094326 29829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:25:02.094928 29829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23: Bootstrap complete.
I20260923 20:25:02.104214 29829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:25:02.105098 29829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64cb309ea8ac47fd8f742dd614321b23, State: Initialized, Role: FOLLOWER
I20260923 20:25:02.105733 29829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:25:02.105964 29829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:25:02.106083 29829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:25:02.106272 29829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:25:02.110100 29829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:25:02.110590 29829 leader_election.cc:302] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64cb309ea8ac47fd8f742dd614321b23; no voters:
I20260923 20:25:02.111322 29829 leader_election.cc:288] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 20:25:02.111465 29833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:25:02.112643 29833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [term 3 LEADER]: Becoming Leader. State: Replica: 64cb309ea8ac47fd8f742dd614321b23, State: Running, Role: LEADER
I20260923 20:25:02.113232 29833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } }
I20260923 20:25:02.113529 29829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:25:02.116035 29835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64cb309ea8ac47fd8f742dd614321b23. Latest consensus state: current_term: 3 leader_uuid: "64cb309ea8ac47fd8f742dd614321b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } } }
I20260923 20:25:02.116510 29835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: This master's current role is: LEADER
I20260923 20:25:02.116034 29834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "64cb309ea8ac47fd8f742dd614321b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64cb309ea8ac47fd8f742dd614321b23" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 43133 } } }
I20260923 20:25:02.116792 29834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64cb309ea8ac47fd8f742dd614321b23 [sys.catalog]: This master's current role is: LEADER
I20260923 20:25:02.127954 29839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:25:02.139508 29839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:25:02.141542 29839 catalog_manager.cc:1259] Loaded cluster ID: c2dff584996248d5b0712946680cf656
I20260923 20:25:02.141721 29839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 20:25:02.144443 29839 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fde3881a9ad google::LogMessage::Fail() at ??:0
@ 0x7fde3881d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fde3881a4de google::LogMessage::Flush() at ??:0
@ 0x7fde3881ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fde4e24d512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fde4e24a976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fde4e2d2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fde4e2d2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fde3b466345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fde3b46991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fde3b469799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fde3b444eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fde3e71e6db start_thread at ??:0
@ 0x7fde376b461f clone at ??:0
W20260923 20:25:02.452466 29831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:02.452762 29831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:02.452862 29831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:02.460243 29831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:02.460355 29831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:02.460466 29831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:25:02.470533 29831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:43327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=41537
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:02.471822 29831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:02.473081 29831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:02.487238 29870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:02.487264 29871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:02.487319 29873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:02.488130 29831 server_base.cc:1034] running on GCE node
I20260923 20:25:02.488724 29831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:02.489634 29831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:02.490836 29831 hybrid_clock.cc:648] HybridClock initialized: now 1790195102490775 us; error 53 us; skew 500 ppm
I20260923 20:25:02.491160 29831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:02 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195102, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:02.494350 29831 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:25:02.495824 29831 webserver.cc:460] Webserver started at http://127.14.127.193:41537/ using document root <none> and password file <none>
I20260923 20:25:02.496527 29831 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:02.496677 29831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:02.502125 29831 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:25:02.507236 29880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:02.508565 29831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 20:25:02.508751 29831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "a378201891c3416eb2da1cded37690c2"
format_stamp: "Formatted at 2026-09-23 20:24:56 on dist-test-slave-wmd0"
server_key: "ba6fdd1e1d06335388c889615f68ce21"
server_key_iv: "df46d48a71cd37f4c8fc5a75b29ec4ac"
server_key_version: "encryptionkey@0"
I20260923 20:25:02.509851 29831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:02.525751 29831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:02.529901 29831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:02.530218 29831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:02.531374 29831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:02.533182 29887 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260923 20:25:02.538928 29831 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 20:25:02.539079 29831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260923 20:25:02.539222 29831 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 20:25:02.542349 29831 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 20:25:02.542460 29831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260923 20:25:02.542904 29887 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Bootstrap starting.
I20260923 20:25:02.551400 29887 log.cc:824] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:02.567167 29887 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:25:02.568056 29887 tablet_bootstrap.cc:492] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Bootstrap complete.
I20260923 20:25:02.568791 29887 ts_tablet_manager.cc:1397] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.007s
I20260923 20:25:02.579841 29887 raft_consensus.cc:348] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:02.581614 29887 raft_consensus.cc:729] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a378201891c3416eb2da1cded37690c2, State: Initialized, Role: FOLLOWER
I20260923 20:25:02.582396 29887 consensus_queue.cc:260] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:02.584066 29887 ts_tablet_manager.cc:1428] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260923 20:25:02.585135 29831 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:43327
I20260923 20:25:02.585222 29995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:43327 every 8 connection(s)
I20260923 20:25:02.586412 29831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:25:02.588937 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 29831
I20260923 20:25:02.589632 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:44095
--local_ip_for_outbound_sockets=127.14.127.194
--tserver_master_addrs=127.14.127.254:43133
--webserver_port=36135
--webserver_interface=127.14.127.194
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:25:02.591662 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:02.589224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:43201)
0923 20:25:02.590134 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:02.590410 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":339,"threads_started":1}
W20260923 20:25:02.592322 29996 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:43133 (0 consecutive failures): Network error: Failed to ping master at 127.14.127.254:43133: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
I20260923 20:25:02.593358 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:02.592856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:39355)
0923 20:25:02.593073 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:02.593176 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:25:02.594451 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:02.593972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:40571)
0923 20:25:02.594170 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:02.594277 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 20:25:02.594755 29996 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:25:02.920943 29999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:02.921227 29999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:02.921363 29999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:02.928526 29999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:02.928629 29999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:02.928740 29999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:25:02.938426 29999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:44095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=36135
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:02.939610 29999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:02.940882 29999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:02.954123 30007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:02.954226 30004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:02.954229 30005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:02.954979 29999 server_base.cc:1034] running on GCE node
I20260923 20:25:02.955574 29999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:02.956493 29999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:02.957713 29999 hybrid_clock.cc:648] HybridClock initialized: now 1790195102957654 us; error 48 us; skew 500 ppm
I20260923 20:25:02.958019 29999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:02 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195102, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:02.961153 29999 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:25:02.962553 29999 webserver.cc:460] Webserver started at http://127.14.127.194:36135/ using document root <none> and password file <none>
I20260923 20:25:02.963248 29999 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:02.963349 29999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:02.968501 29999 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:25:02.971292 30014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:02.972599 29999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:25:02.972733 29999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "e4b5fffa323347e68c3346ec5867f30a"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "aff44576c715fb8b81e4b3c8f82b2bbf"
server_key_iv: "c7506c1c2aaf3d3183890d59b2af2787"
server_key_version: "encryptionkey@0"
I20260923 20:25:02.973651 29999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:02.985371 29999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:02.989565 29999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:02.989785 29999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:02.990777 29999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:02.992041 29999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:02.992129 29999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:02.992199 29999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:02.992265 29999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:03.025676 29999 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:44095
I20260923 20:25:03.025774 30126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:44095 every 8 connection(s)
I20260923 20:25:03.027235 29999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:25:03.031921 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:03.030127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:41239)
0923 20:25:03.030873 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:03.031090 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":268,"threads_started":1}
W20260923 20:25:03.032593 30127 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:43133 (0 consecutive failures): Network error: Failed to ping master at 127.14.127.254:43133: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
I20260923 20:25:03.033337 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:03.033013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:52067)
0923 20:25:03.033150 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:03.033234 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 20:25:03.033640 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 29999
I20260923 20:25:03.034207 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:03.033887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:57973)
0923 20:25:03.034023 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:03.034102 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 20:25:03.034237 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:42243
--local_ip_for_outbound_sockets=127.14.127.195
--tserver_master_addrs=127.14.127.254:43133
--webserver_port=41823
--webserver_interface=127.14.127.195
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:25:03.034483 30127 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:25:03.359681 30130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:03.359949 30130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:03.360121 30130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:03.367401 30130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:03.367547 30130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:03.367733 30130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:25:03.377621 30130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:42243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=41823
--tserver_master_addrs=127.14.127.254:43133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:03.378953 30130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:03.380280 30130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:03.393743 30135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:03.394662 30136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:03.394891 30138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:03.395946 30130 server_base.cc:1034] running on GCE node
I20260923 20:25:03.396557 30130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:03.397472 30130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:03.398712 30130 hybrid_clock.cc:648] HybridClock initialized: now 1790195103398655 us; error 47 us; skew 500 ppm
I20260923 20:25:03.399046 30130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:03 dist-test-slave-wmd0 krb5kdc[28685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195103, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:03.402163 30130 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:25:03.403674 30130 webserver.cc:460] Webserver started at http://127.14.127.195:41823/ using document root <none> and password file <none>
I20260923 20:25:03.404431 30130 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:03.404583 30130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:03.410061 30130 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:25:03.412833 30145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:03.414538 30130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:25:03.414762 30130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "2a0b567a59e249ae8eac7c10e0001e46"
format_stamp: "Formatted at 2026-09-23 20:24:57 on dist-test-slave-wmd0"
server_key: "3269b677c6e10d8ded3ae7e766db1371"
server_key_iv: "6e8d3ee9ca71b09ba53c89455db719b5"
server_key_version: "encryptionkey@0"
I20260923 20:25:03.415791 30130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:03.434477 30130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:03.438894 30130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:03.439167 30130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:03.440203 30130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:03.441548 30130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:03.441672 30130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:03.441814 30130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:03.441905 30130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:03.475658 30130 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:42243
I20260923 20:25:03.475728 30257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:42243 every 8 connection(s)
I20260923 20:25:03.477324 30130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:25:03.482062 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:03.480255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:58977)
0923 20:25:03.480959 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:03.481173 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":269,"threads_started":1}
W20260923 20:25:03.482892 30258 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:43133 (0 consecutive failures): Network error: Failed to ping master at 127.14.127.254:43133: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
I20260923 20:25:03.483749 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:03.483434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:51299)
0923 20:25:03.483569 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:03.483637 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:25:03.484583 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:03.484256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:52365)
0923 20:25:03.484397 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:03.484479 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:25:03.484810 30258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:25:03.487638 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 30130
I20260923 20:25:03.595968 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:03.595260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:41493)
0923 20:25:03.595671 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:03.595759 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":196,"threads_started":1}
I20260923 20:25:04.034480 30263 raft_consensus.cc:482] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:25:04.034875 30263 raft_consensus.cc:504] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:04.037120 30263 leader_election.cc:288] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
I20260923 20:25:04.046468 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:04.035154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:33093)
0923 20:25:04.045938 (+ 10784us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:04.046128 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10643,"threads_started":1}
Sep 23 20:25:04 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195102, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
Sep 23 20:25:04 dist-test-slave-wmd0 krb5kdc[28685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195102, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:25:04.063963 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:04.037275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:42243 (local address 127.14.127.193:46121)
0923 20:25:04.038252 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:04.038286 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:04.038878 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:04.044658 (+ 5780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:04.044674 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:04.045766 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:04.046649 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:04.046679 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:04.049514 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:04.049522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:04.051485 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:04.051499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:04.052936 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:04.052940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:04.053116 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:04.053819 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:04.053841 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:04.056542 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:04.060464 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:04.060470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:04.060481 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:04.061016 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:04.061526 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:04.061529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:04.061531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:04.061693 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:04.062785 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:04.062790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:04.063039 (+ 249us) client_negotiation.cc:769] Sending connection context
0923 20:25:04.063270 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 20:25:04.063601 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819}
I20260923 20:25:04.064445 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:04.038528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:46121 (local address 127.14.127.195:42243)
0923 20:25:04.039222 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:04.039244 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:04.039297 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:25:04.042277 (+ 2980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:04.042293 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:04.043437 (+ 1144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:04.043894 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:04.047122 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:04.049272 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:04.051639 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:04.052740 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:04.052941 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:04.056880 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:04.056890 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:04.056909 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:04.056963 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:04.060334 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:04.061173 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:04.061184 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:04.061194 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:04.061411 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:04.061836 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:04.061843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:04.061846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:04.062608 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:04.062844 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:04.063563 (+ 719us) server_negotiation.cc:299] Negotiation successful
0923 20:25:04.063995 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
I20260923 20:25:04.066843 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:04.037581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:44095 (local address 127.14.127.193:51229)
0923 20:25:04.041645 (+ 4064us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:04.041660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:04.041749 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:04.051774 (+ 10025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:04.051777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:04.052119 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:04.052429 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:04.052445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:04.054328 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:04.054334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:04.055871 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:04.055891 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:04.056852 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:04.056857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:04.056977 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:04.057568 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:04.057592 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:04.059764 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:04.063884 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:04.063891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:04.063895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:04.064338 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:04.064828 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:04.064834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:04.064837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:04.065026 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:04.065829 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:04.065836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:04.065999 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 20:25:04.066164 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 20:25:04.066469 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3955,"thread_start_us":304,"threads_started":1}
I20260923 20:25:04.067618 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:04.038039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:51229 (local address 127.14.127.194:44095)
0923 20:25:04.048718 (+ 10679us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:04.048736 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:04.048782 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:25:04.051055 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:04.051067 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:04.051485 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:04.051861 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:04.052617 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:04.054149 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:04.056003 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:04.056684 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:04.056874 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:04.060056 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:04.060067 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:04.060085 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:04.060133 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:04.063743 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:04.064484 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:04.064491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:04.064498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:04.064681 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:04.065151 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:04.065154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:04.065156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:04.065672 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:04.065874 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:04.066794 (+ 920us) server_negotiation.cc:299] Negotiation successful
0923 20:25:04.067239 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10472,"thread_start_us":12592,"threads_started":1}
W20260923 20:25:04.068037 29882 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
W20260923 20:25:04.070163 29881 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
I20260923 20:25:04.070287 29881 leader_election.cc:302] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a378201891c3416eb2da1cded37690c2; no voters: 2a0b567a59e249ae8eac7c10e0001e46, e4b5fffa323347e68c3346ec5867f30a
I20260923 20:25:04.070820 30263 raft_consensus.cc:2726] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:25:04.486395 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:04.485337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:37559)
0923 20:25:04.485892 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:04.486034 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":295,"threads_started":1}
I20260923 20:25:04.597723 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:04.597000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:42039)
0923 20:25:04.597439 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:04.597529 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":229,"threads_started":1}
I20260923 20:25:05.048328 30270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:05.047544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:37217)
0923 20:25:05.048093 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:05.048174 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":334,"threads_started":1}
I20260923 20:25:05.488132 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:05.487421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:51255)
0923 20:25:05.487872 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:05.487954 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260923 20:25:05.599649 30272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:05.598808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:39771)
0923 20:25:05.599301 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:05.599408 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":224,"threads_started":1}
I20260923 20:25:05.672910 30273 raft_consensus.cc:482] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:25:05.673147 30273 raft_consensus.cc:504] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:05.673853 30273 leader_election.cc:288] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
W20260923 20:25:05.674852 29882 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
W20260923 20:25:05.675014 29881 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
I20260923 20:25:05.675172 29881 leader_election.cc:302] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a378201891c3416eb2da1cded37690c2; no voters: 2a0b567a59e249ae8eac7c10e0001e46, e4b5fffa323347e68c3346ec5867f30a
I20260923 20:25:05.675403 30273 raft_consensus.cc:2726] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:25:06.050112 30274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:06.049273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:47937)
0923 20:25:06.049835 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:06.049935 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":301,"threads_started":1}
I20260923 20:25:06.489710 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:06.488986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:57995)
0923 20:25:06.489450 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:06.489533 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":242,"threads_started":1}
I20260923 20:25:06.601352 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:06.600573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:37881)
0923 20:25:06.601059 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:06.601172 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":278,"threads_started":1}
I20260923 20:25:07.051925 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:07.051125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:50505)
0923 20:25:07.051589 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:07.051680 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
I20260923 20:25:07.491365 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:07.490707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:57335)
0923 20:25:07.491111 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:07.491188 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":198,"threads_started":1}
I20260923 20:25:07.603224 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:07.602297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:39487)
0923 20:25:07.602912 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:07.603021 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":232,"threads_started":1}
I20260923 20:25:08.053562 30280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:08.052854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:54291)
0923 20:25:08.053321 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:08.053402 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":255,"threads_started":1}
I20260923 20:25:08.493278 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:08.492502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:54569)
0923 20:25:08.493022 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:08.493107 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":283,"threads_started":1}
I20260923 20:25:08.551816 30282 raft_consensus.cc:482] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:25:08.551997 30282 raft_consensus.cc:504] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:08.552603 30282 leader_election.cc:288] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
W20260923 20:25:08.553508 29881 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
W20260923 20:25:08.553712 29882 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
I20260923 20:25:08.553830 29882 leader_election.cc:302] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a378201891c3416eb2da1cded37690c2; no voters: 2a0b567a59e249ae8eac7c10e0001e46, e4b5fffa323347e68c3346ec5867f30a
I20260923 20:25:08.554162 30282 raft_consensus.cc:2726] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:25:08.604920 30283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:08.604136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:56053)
0923 20:25:08.604625 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:08.604735 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":245,"threads_started":1}
I20260923 20:25:09.055471 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:09.054664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:57195)
0923 20:25:09.055204 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:09.055292 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
I20260923 20:25:09.495023 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:09.494275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:60289)
0923 20:25:09.494763 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:09.494849 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":234,"threads_started":1}
I20260923 20:25:09.606519 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:09.605814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:48697)
0923 20:25:09.606257 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:09.606338 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260923 20:25:10.057457 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:10.056422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:35895)
0923 20:25:10.057052 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:10.057179 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":319,"threads_started":1}
I20260923 20:25:10.496762 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:10.495975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:53337)
0923 20:25:10.496489 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:10.496575 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":232,"threads_started":1}
I20260923 20:25:10.608237 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:10.607511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:50779)
0923 20:25:10.607959 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:10.608043 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260923 20:25:11.059139 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:11.058388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:35463)
0923 20:25:11.058880 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:11.058963 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":291,"threads_started":1}
I20260923 20:25:11.498521 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:11.497723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:34483)
0923 20:25:11.498238 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:11.498333 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":258,"threads_started":1}
I20260923 20:25:11.609926 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:11.609234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:58011)
0923 20:25:11.609671 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:11.609755 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":232,"threads_started":1}
I20260923 20:25:11.796802 30293 raft_consensus.cc:482] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:25:11.797048 30293 raft_consensus.cc:504] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:11.797797 30293 leader_election.cc:288] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
W20260923 20:25:11.798748 29882 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
W20260923 20:25:11.798921 29881 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
I20260923 20:25:11.799088 29881 leader_election.cc:302] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a378201891c3416eb2da1cded37690c2; no voters: 2a0b567a59e249ae8eac7c10e0001e46, e4b5fffa323347e68c3346ec5867f30a
I20260923 20:25:11.799350 30293 raft_consensus.cc:2726] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:25:12.060850 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:12.060068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:54981)
0923 20:25:12.060587 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:12.060689 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":247,"threads_started":1}
I20260923 20:25:12.500293 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:12.499548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:60853)
0923 20:25:12.500049 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:12.500140 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":241,"threads_started":1}
I20260923 20:25:12.611670 30296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:12.610969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:50511)
0923 20:25:12.611414 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:12.611496 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260923 20:25:13.062762 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:13.061894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:48657)
0923 20:25:13.062396 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:13.062481 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":292,"threads_started":1}
I20260923 20:25:13.501964 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:13.501257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:43325)
0923 20:25:13.501697 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:13.501786 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260923 20:25:13.613227 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:13.612545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:59839)
0923 20:25:13.612981 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:13.613061 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":230,"threads_started":1}
I20260923 20:25:14.064415 30300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:14.063623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:43379)
0923 20:25:14.064178 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:14.064258 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":310,"threads_started":1}
I20260923 20:25:14.503665 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:14.502899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:58891)
0923 20:25:14.503409 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:14.503500 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":269,"threads_started":1}
I20260923 20:25:14.614892 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:14.614058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:34083)
0923 20:25:14.614585 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:14.614729 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":311,"threads_started":1}
I20260923 20:25:15.066160 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:15.065442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:48127)
0923 20:25:15.065890 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:15.065984 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":232,"threads_started":1}
I20260923 20:25:15.505438 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:15.504686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:51717)
0923 20:25:15.505183 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:15.505269 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":251,"threads_started":1}
I20260923 20:25:15.616492 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:15.615726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:47993)
0923 20:25:15.616206 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:15.616295 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":265,"threads_started":1}
I20260923 20:25:16.067981 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:16.067191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:33801)
0923 20:25:16.067694 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:16.067790 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":257,"threads_started":1}
I20260923 20:25:16.507308 30307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:16.506537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:40215)
0923 20:25:16.507052 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:16.507149 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":290,"threads_started":1}
I20260923 20:25:16.618449 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:16.617432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:43321)
0923 20:25:16.618166 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:16.618245 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":519,"threads_started":1}
I20260923 20:25:17.069664 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:17.068919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:44683)
0923 20:25:17.069363 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:17.069459 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260923 20:25:17.509330 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:17.508412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:54003)
0923 20:25:17.508940 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:17.509040 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":290,"threads_started":1}
I20260923 20:25:17.620617 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:17.619753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.193:42701)
0923 20:25:17.620248 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:17.620365 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":324,"threads_started":1}
I20260923 20:25:17.991483 30312 raft_consensus.cc:482] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:25:17.991686 30312 raft_consensus.cc:504] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a378201891c3416eb2da1cded37690c2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 43327 } } peers { permanent_uuid: "2a0b567a59e249ae8eac7c10e0001e46" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 42243 } } peers { permanent_uuid: "e4b5fffa323347e68c3346ec5867f30a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 44095 } }
I20260923 20:25:17.992393 30312 leader_election.cc:288] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243), e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095)
W20260923 20:25:17.993420 29882 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a0b567a59e249ae8eac7c10e0001e46 (127.14.127.195:42243): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
W20260923 20:25:17.993592 29881 leader_election.cc:341] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4b5fffa323347e68c3346ec5867f30a (127.14.127.194:44095): Not found: Tablet not found: 462c684450f04f42b96fa91fb60a61e5
I20260923 20:25:17.993758 29881 leader_election.cc:302] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a378201891c3416eb2da1cded37690c2; no voters: 2a0b567a59e249ae8eac7c10e0001e46, e4b5fffa323347e68c3346ec5867f30a
I20260923 20:25:17.994021 30312 raft_consensus.cc:2726] T 462c684450f04f42b96fa91fb60a61e5 P a378201891c3416eb2da1cded37690c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:25:18.071527 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:18.070827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.194:34519)
0923 20:25:18.071261 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:18.071365 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260923 20:25:18.488862 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 29831
I20260923 20:25:18.510505 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:18.489979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.14.127.193:43327)
0923 20:25:18.490674 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:18.490684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:18.490715 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:25:18.490788 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:18.490797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:18.491080 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:18.491286 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:18.492920 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:18.493942 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:18.495477 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:18.496049 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:18.496149 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:18.504744 (+ 8595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:18.504759 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:18.504777 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:18.504819 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:18.507737 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:18.508481 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:18.508489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:18.508491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:18.508625 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:18.508971 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:18.508973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:18.508975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:18.509342 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:18.509556 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:18.509899 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 20:25:18.510200 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":342,"threads_started":1}
I20260923 20:25:18.511425 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:18.510516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:43133 (local address 127.14.127.195:42161)
0923 20:25:18.511161 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:18.511287 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":379,"threads_started":1}
I20260923 20:25:18.511933 29990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:18.603130 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 29831
I20260923 20:25:18.621829 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 29999
I20260923 20:25:18.635726 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:18.622672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.14.127.194:44095)
0923 20:25:18.623208 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:18.623212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:18.623227 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:25:18.623282 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:18.623286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:18.623344 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:18.623470 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:18.624890 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:18.625648 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:18.627103 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:18.627559 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:18.627696 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:18.630384 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:18.630389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:18.630393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:18.630425 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:18.632999 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:18.633700 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:18.633709 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:18.633714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:18.633854 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:18.634240 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:18.634245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:18.634248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:18.634535 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:18.634851 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:18.635127 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 20:25:18.635349 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":232,"threads_started":1}
I20260923 20:25:18.636400 30122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:18.720311 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 29999
I20260923 20:25:18.738672 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 30130
I20260923 20:25:18.752221 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:18.739543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.14.127.195:42243)
0923 20:25:18.740161 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:18.740166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:18.740184 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:25:18.740239 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:18.740243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:18.740311 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:18.740402 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:18.741814 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:18.742695 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:18.743856 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:18.744264 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:18.744351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:18.747092 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:18.747097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:18.747100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:18.747132 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:18.749678 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:18.750323 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:18.750327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:18.750329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:18.750470 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:18.750849 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:18.750852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:18.750854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:18.751104 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:18.751309 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:18.751621 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 20:25:18.751815 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":340,"threads_started":1}
I20260923 20:25:18.752907 30253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:18.836474 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 30130
2026-09-23T20:25:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24282 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:25:18 dist-test-slave-wmd0 krb5kdc[30323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:25:18 dist-test-slave-wmd0 krb5kdc[30323](info): set up 2 sockets
Sep 23 20:25:18 dist-test-slave-wmd0 krb5kdc[30323](info): commencing operation
krb5kdc: starting...
W20260923 20:25:19.939275 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:25:20 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:25:20.178077 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:32835
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:25:20.518304 30336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:20.518646 30336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:20.518802 30336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:20.526485 30336 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:25:20.526664 30336 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:25:20.526758 30336 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:25:20.526873 30336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:25:20.526968 30336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:20.527101 30336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:20.527164 30336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:25:20.527261 30336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:25:20.537634 30336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:20.539040 30336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:20.540450 30336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:20.552139 30341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:20.552095 30344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:20.552174 30342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:20.553382 30336 server_base.cc:1034] running on GCE node
I20260923 20:25:20.554134 30336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:20.555529 30336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:20.556800 30336 hybrid_clock.cc:648] HybridClock initialized: now 1790195120556730 us; error 65 us; skew 500 ppm
I20260923 20:25:20.557133 30336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:20 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195120, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:20.560179 30336 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:25:20.561502 30336 webserver.cc:460] Webserver started at http://127.14.127.254:35547/ using document root <none> and password file <none>
I20260923 20:25:20.562150 30336 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:20.562232 30336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:20.562479 30336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:25:20.565389 30336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "8a4a0a2999b44936ba4d9b04f37f9eec"
format_stamp: "Formatted at 2026-09-23 20:25:20 on dist-test-slave-wmd0"
server_key: "ee42adf781353a2627a4668e08683a25"
server_key_iv: "9c40da1e8c197da3fdeb85426833b893"
server_key_version: "encryptionkey@0"
I20260923 20:25:20.566128 30336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "8a4a0a2999b44936ba4d9b04f37f9eec"
format_stamp: "Formatted at 2026-09-23 20:25:20 on dist-test-slave-wmd0"
server_key: "ee42adf781353a2627a4668e08683a25"
server_key_iv: "9c40da1e8c197da3fdeb85426833b893"
server_key_version: "encryptionkey@0"
I20260923 20:25:20.571329 30336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:25:20.574683 30351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:20.576092 30336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:25:20.576236 30336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "8a4a0a2999b44936ba4d9b04f37f9eec"
format_stamp: "Formatted at 2026-09-23 20:25:20 on dist-test-slave-wmd0"
server_key: "ee42adf781353a2627a4668e08683a25"
server_key_iv: "9c40da1e8c197da3fdeb85426833b893"
server_key_version: "encryptionkey@0"
I20260923 20:25:20.576414 30336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:20.620299 30336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:20.624370 30336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:20.624617 30336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:20.643445 30336 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:32835
I20260923 20:25:20.643460 30402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:32835 every 8 connection(s)
I20260923 20:25:20.644812 30336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:25:20.648550 30403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:20.651710 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 30336
I20260923 20:25:20.651937 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:25:20.652657 14847 external_mini_cluster.cc:1428] Setting key c46887ddab1f100c0d8e4ca42242100f
I20260923 20:25:20.661998 30403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Bootstrap starting.
I20260923 20:25:20.665831 30403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:20.666962 30403 log.cc:824] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:25:20 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:20.669974 30403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: No bootstrap required, opened a new log
I20260923 20:25:20.677134 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:20.654694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.14.127.254:32835)
0923 20:25:20.655545 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:20.655562 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:20.655615 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:25:20.657715 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:20.657750 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:20.658312 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:20.658808 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:20.660218 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:20.662206 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:20.663748 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:20.664797 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:20.664996 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:20.669616 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:20.669624 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:20.669639 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:20.669674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:20.673010 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:20.673759 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:20.673765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:20.673771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:20.673941 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:20.674353 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:20.674357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:20.674359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:20.674963 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:20.675295 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:20.675733 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 20:25:20.676147 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":340,"threads_started":1}
I20260923 20:25:20.678198 30403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:25:20.678571 30403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:20.678735 30403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4a0a2999b44936ba4d9b04f37f9eec, State: Initialized, Role: FOLLOWER
I20260923 20:25:20.679358 30403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:25:20.679705 30403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:25:20.679868 30403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:25:20.680078 30403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:20.682264 30403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:25:20.682821 30403 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a4a0a2999b44936ba4d9b04f37f9eec; no voters:
I20260923 20:25:20.683422 30403 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:25:20.683565 30408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:25:20.684258 30408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 1 LEADER]: Becoming Leader. State: Replica: 8a4a0a2999b44936ba4d9b04f37f9eec, State: Running, Role: LEADER
I20260923 20:25:20.684871 30408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:25:20.685500 30403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:25:20.687642 30410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4a0a2999b44936ba4d9b04f37f9eec. Latest consensus state: current_term: 1 leader_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } } }
I20260923 20:25:20.688012 30410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: This master's current role is: LEADER
I20260923 20:25:20.687872 30409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } } }
I20260923 20:25:20.688520 30409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: This master's current role is: LEADER
I20260923 20:25:20.692006 30413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:25:20.697614 30413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:25:20.706393 30413 catalog_manager.cc:1347] Generated new cluster ID: eaef0f26cbd5438790cd497dff794cff
I20260923 20:25:20.706529 30413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:25:20.725142 30413 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:25:20.727056 30413 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:25:20.739511 30413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Generated new TSK 0
I20260923 20:25:20.740360 30413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:25:20.806025 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:32835
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:25:21.137828 30430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:21.138110 30430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:21.138273 30430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:21.145478 30430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:21.145588 30430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:21.145776 30430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:25:21.155725 30430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:21.157070 30430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:21.158368 30430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:21.171579 30436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:21.171625 30435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:21.171600 30438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:21.172783 30430 server_base.cc:1034] running on GCE node
I20260923 20:25:21.173326 30430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:21.174202 30430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:21.175407 30430 hybrid_clock.cc:648] HybridClock initialized: now 1790195121175352 us; error 56 us; skew 500 ppm
I20260923 20:25:21.175716 30430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:21 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195121, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:21.178958 30430 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:25:21.180449 30430 webserver.cc:460] Webserver started at http://127.14.127.193:36609/ using document root <none> and password file <none>
I20260923 20:25:21.181154 30430 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:21.181308 30430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:21.181615 30430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:25:21.184404 30430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "9055e0d5f7fd4c2ebf792773d9b95e15"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "ad4bb5c5c5309bbdfde13e821dde93b7"
server_key_iv: "11f847fd3465290f8d937728ad96e70d"
server_key_version: "encryptionkey@0"
I20260923 20:25:21.185238 30430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "9055e0d5f7fd4c2ebf792773d9b95e15"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "ad4bb5c5c5309bbdfde13e821dde93b7"
server_key_iv: "11f847fd3465290f8d937728ad96e70d"
server_key_version: "encryptionkey@0"
I20260923 20:25:21.191015 30430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 20:25:21.196108 30445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:21.197763 30430 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:25:21.197965 30430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "9055e0d5f7fd4c2ebf792773d9b95e15"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "ad4bb5c5c5309bbdfde13e821dde93b7"
server_key_iv: "11f847fd3465290f8d937728ad96e70d"
server_key_version: "encryptionkey@0"
I20260923 20:25:21.198197 30430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:21.228199 30430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:21.232579 30430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:21.232810 30430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:21.233706 30430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:21.234934 30430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:21.235004 30430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:21.235092 30430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:21.235142 30430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:21.268324 30430 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:33351
I20260923 20:25:21.268371 30557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:33351 every 8 connection(s)
I20260923 20:25:21.269647 30430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:25:21.279654 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 30430
I20260923 20:25:21.279906 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:25:21.280683 14847 external_mini_cluster.cc:1428] Setting key 87619fefef1ab197d7cb14a837f4b99d
Sep 23 20:25:21 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195121, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:21.290449 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:21.272289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:50191 (local address 127.14.127.254:32835)
0923 20:25:21.272757 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:21.272762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:21.273833 (+ 1071us) server_negotiation.cc:407] Connection header received
0923 20:25:21.275787 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:21.275792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:21.275886 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:21.276010 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:21.277500 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:21.278274 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:21.279949 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:21.280407 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:21.280549 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:21.284408 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:21.284412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:21.284416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:21.284447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:21.287436 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:21.288161 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:21.288171 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:21.288174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:21.288287 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:21.288727 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:21.288732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:21.288735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:21.288996 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:21.289150 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:21.289859 (+ 709us) server_negotiation.cc:299] Negotiation successful
0923 20:25:21.290084 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260923 20:25:21.291477 30561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:21.272864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:50191)
0923 20:25:21.273573 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:21.273611 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:21.275487 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:21.276263 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:21.276271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:21.276784 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:21.277328 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:21.277344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:21.278454 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:21.278458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:21.279766 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:21.279782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:21.280651 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:21.280659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:21.280960 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:21.281720 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:21.281743 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:21.284192 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:21.287575 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:21.287585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:21.287601 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:21.288028 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:21.288397 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:21.288403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:21.288407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:21.288602 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:21.289125 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:21.289132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:21.289478 (+ 346us) client_negotiation.cc:769] Sending connection context
0923 20:25:21.289740 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 20:25:21.290282 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":310,"threads_started":1}
I20260923 20:25:21.293083 30558 heartbeater.cc:347] Connected to a master server at 127.14.127.254:32835
I20260923 20:25:21.293418 30558 heartbeater.cc:464] Registering TS with master...
I20260923 20:25:21.294200 30558 heartbeater.cc:511] Master 127.14.127.254:32835 requested a full tablet report, sending...
I20260923 20:25:21.296547 30368 ts_manager.cc:194] Registered new tserver with Master: 9055e0d5f7fd4c2ebf792773d9b95e15 (127.14.127.193:33351)
I20260923 20:25:21.298777 30368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:50191
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:25:21.332019 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:32835
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:25:21.651180 30564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:21.651474 30564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:21.651655 30564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:21.658851 30564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:21.658991 30564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:21.659175 30564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:25:21.668746 30564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:21.670063 30564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:21.671485 30564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:21.685838 30569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:21.685786 30572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:21.685762 30570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:21.686545 30564 server_base.cc:1034] running on GCE node
I20260923 20:25:21.687475 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:21.688482 30564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:21.689713 30564 hybrid_clock.cc:648] HybridClock initialized: now 1790195121689663 us; error 59 us; skew 500 ppm
I20260923 20:25:21.690022 30564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:21 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195121, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:21.693475 30564 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:25:21.694999 30564 webserver.cc:460] Webserver started at http://127.14.127.194:43273/ using document root <none> and password file <none>
I20260923 20:25:21.695650 30564 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:21.695797 30564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:21.696113 30564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:25:21.699070 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "6949356cbfb44ef8b0aeea63a43bf49e"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "69f8f3b786b250ac60c7dff87f136b7a"
server_key_iv: "25b14fb9333d59abc423d5cd0e84b6ab"
server_key_version: "encryptionkey@0"
I20260923 20:25:21.699828 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "6949356cbfb44ef8b0aeea63a43bf49e"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "69f8f3b786b250ac60c7dff87f136b7a"
server_key_iv: "25b14fb9333d59abc423d5cd0e84b6ab"
server_key_version: "encryptionkey@0"
I20260923 20:25:21.705332 30564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 20:25:21.708755 30579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:21.709941 30564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 20:25:21.710109 30564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "6949356cbfb44ef8b0aeea63a43bf49e"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "69f8f3b786b250ac60c7dff87f136b7a"
server_key_iv: "25b14fb9333d59abc423d5cd0e84b6ab"
server_key_version: "encryptionkey@0"
I20260923 20:25:21.710336 30564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:21.721060 30564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:21.725383 30564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:21.725692 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:21.726631 30564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:21.727929 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:21.728000 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:21.728173 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:21.728224 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:21.760952 30564 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:34589
I20260923 20:25:21.760998 30691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:34589 every 8 connection(s)
I20260923 20:25:21.762248 30564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:25:21.763489 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 30564
I20260923 20:25:21.763703 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:25:21.764254 14847 external_mini_cluster.cc:1428] Setting key 43d2d99dac987a864aedf5d255394150
Sep 23 20:25:21 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195121, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:21.783046 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:21.765206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:41391 (local address 127.14.127.254:32835)
0923 20:25:21.765383 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:21.765387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:21.766792 (+ 1405us) server_negotiation.cc:407] Connection header received
0923 20:25:21.768623 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:21.768628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:21.768726 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:21.768832 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:21.770282 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:21.771032 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:21.772544 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:21.772963 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:21.773098 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:21.777062 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:21.777067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:21.777070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:21.777102 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:21.779768 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:21.780662 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:21.780668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:21.780672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:21.780808 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:21.781231 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:21.781239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:21.781244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:21.781524 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:21.781683 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:21.782427 (+ 744us) server_negotiation.cc:299] Negotiation successful
0923 20:25:21.782687 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:25:21.784015 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:21.765749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:41391)
0923 20:25:21.766583 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:21.766663 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:21.768362 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:21.769063 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:21.769072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:21.769570 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:21.770103 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:21.770119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:21.771177 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:21.771183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:21.772385 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:21.772401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:21.773195 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:21.773202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:21.773423 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:21.774116 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:21.774137 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:21.776840 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:21.779908 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:21.779917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:21.779934 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:21.780542 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:21.780923 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:21.780928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:21.780932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:21.781117 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:21.781656 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:21.781664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:21.782032 (+ 368us) client_negotiation.cc:769] Sending connection context
0923 20:25:21.782325 (+ 293us) client_negotiation.cc:240] Negotiation successful
0923 20:25:21.782872 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":295,"threads_started":1}
I20260923 20:25:21.785472 30692 heartbeater.cc:347] Connected to a master server at 127.14.127.254:32835
I20260923 20:25:21.785854 30692 heartbeater.cc:464] Registering TS with master...
I20260923 20:25:21.786760 30692 heartbeater.cc:511] Master 127.14.127.254:32835 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
I20260923 20:25:21.788369 30368 ts_manager.cc:194] Registered new tserver with Master: 6949356cbfb44ef8b0aeea63a43bf49e (127.14.127.194:34589)
I20260923 20:25:21.789120 30368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:41391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:25:21.815626 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:32835
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:25:22.136934 30697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:22.137202 30697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:22.137362 30697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:22.144742 30697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:22.144886 30697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:22.145082 30697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:25:22.155287 30697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:22.156560 30697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:22.157923 30697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:22.170962 30702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:22.170979 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:22.170965 30703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:22.171775 30697 server_base.cc:1034] running on GCE node
I20260923 20:25:22.172385 30697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:22.173345 30697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:22.174556 30697 hybrid_clock.cc:648] HybridClock initialized: now 1790195122174486 us; error 67 us; skew 500 ppm
I20260923 20:25:22.174950 30697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:22 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:22.178084 30697 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:25:22.179646 30697 webserver.cc:460] Webserver started at http://127.14.127.195:40377/ using document root <none> and password file <none>
I20260923 20:25:22.180315 30697 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:22.180449 30697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:22.180768 30697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:25:22.183766 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "4bd98534c84c49e985134570c14c8b21"
format_stamp: "Formatted at 2026-09-23 20:25:22 on dist-test-slave-wmd0"
server_key: "24fdf737ac3b233fcf00aa46d3c400a9"
server_key_iv: "1acce4ce8024d3f19fd0f33962639a86"
server_key_version: "encryptionkey@0"
I20260923 20:25:22.184514 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "4bd98534c84c49e985134570c14c8b21"
format_stamp: "Formatted at 2026-09-23 20:25:22 on dist-test-slave-wmd0"
server_key: "24fdf737ac3b233fcf00aa46d3c400a9"
server_key_iv: "1acce4ce8024d3f19fd0f33962639a86"
server_key_version: "encryptionkey@0"
I20260923 20:25:22.189774 30697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:25:22.193320 30712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:22.194828 30697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:25:22.195019 30697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "4bd98534c84c49e985134570c14c8b21"
format_stamp: "Formatted at 2026-09-23 20:25:22 on dist-test-slave-wmd0"
server_key: "24fdf737ac3b233fcf00aa46d3c400a9"
server_key_iv: "1acce4ce8024d3f19fd0f33962639a86"
server_key_version: "encryptionkey@0"
I20260923 20:25:22.195250 30697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:22.206959 30697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:22.210839 30697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:22.211122 30697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:22.212018 30697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:22.213291 30697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:22.213393 30697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:22.213537 30697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:22.213615 30697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:22.246479 30697 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:35885
I20260923 20:25:22.246520 30824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:35885 every 8 connection(s)
I20260923 20:25:22.247833 30697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:25:22.257128 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 30697
I20260923 20:25:22.257431 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:25:22.257908 14847 external_mini_cluster.cc:1428] Setting key 0ed7dd1d86110915e52a806cf9ee2a83
Sep 23 20:25:22 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195122, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:22.267511 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.250290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:41485 (local address 127.14.127.254:32835)
0923 20:25:22.250516 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.250522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.251724 (+ 1202us) server_negotiation.cc:407] Connection header received
0923 20:25:22.253436 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.253441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.253527 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.253619 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:22.255131 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.255902 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.257424 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.257903 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.258039 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.262042 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:22.262047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:22.262050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:22.262082 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:22.264463 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.265165 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.265176 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.265178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.265302 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.265721 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.265732 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.265734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.266004 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:22.266184 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.266894 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.267133 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:25:22.268498 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.250864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:41485)
0923 20:25:22.251554 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:22.251597 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:22.253187 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:22.253812 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:22.253820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:22.254343 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:22.254978 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.254995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.256035 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.256039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.257276 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.257287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.258161 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.258170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.258431 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.259284 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:22.259314 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:22.261846 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:22.264602 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:22.264611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:22.264626 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:22.265028 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:22.265411 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:22.265417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:22.265420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:22.265600 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:22.266160 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:22.266168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:22.266483 (+ 315us) client_negotiation.cc:769] Sending connection context
0923 20:25:22.266780 (+ 297us) client_negotiation.cc:240] Negotiation successful
0923 20:25:22.267316 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":273,"threads_started":1}
I20260923 20:25:22.269966 30825 heartbeater.cc:347] Connected to a master server at 127.14.127.254:32835
I20260923 20:25:22.270272 30825 heartbeater.cc:464] Registering TS with master...
I20260923 20:25:22.270973 30825 heartbeater.cc:511] Master 127.14.127.254:32835 requested a full tablet report, sending...
I20260923 20:25:22.272559 30368 ts_manager.cc:194] Registered new tserver with Master: 4bd98534c84c49e985134570c14c8b21 (127.14.127.195:35885)
I20260923 20:25:22.273645 30368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:41485
I20260923 20:25:22.274444 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:25:22.291528 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.279095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.14.127.254:32835)
0923 20:25:22.279310 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.279318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.279652 (+ 334us) server_negotiation.cc:407] Connection header received
0923 20:25:22.279835 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.279842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.279906 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.280005 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:22.281446 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.282155 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.283529 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.283942 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.284080 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.286400 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:22.286405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:22.286408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:22.286439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:22.288870 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.289546 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.289554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.289557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.289701 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.290065 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.290071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.290074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.290382 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:22.290666 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.290987 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.291207 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:25:22.295912 30367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 20:25:22.302178 30558 heartbeater.cc:503] Master 127.14.127.254:32835 was elected leader, sending a full tablet report...
W20260923 20:25:22.303272 30367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:25:22.326136 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.315596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.14.127.194:34589)
0923 20:25:22.316269 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.316279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.316727 (+ 448us) server_negotiation.cc:407] Connection header received
0923 20:25:22.316834 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.316841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.317125 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.317307 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:22.318317 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.320251 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.323457 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.324924 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.325102 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.325308 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.325614 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.325790 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":934,"threads_started":1}
I20260923 20:25:22.326182 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.315533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:34589 (local address 127.0.0.1:38402)
0923 20:25:22.316295 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:22.316349 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:22.316543 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:22.317583 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:22.317592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:22.317622 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:22.318072 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.318091 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.320412 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.320416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.323267 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.323281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.325156 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.325162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.325359 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.325410 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:25:22.325551 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 20:25:22.325770 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":335,"threads_started":1}
I20260923 20:25:22.327809 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.316076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:35885 (local address 127.0.0.1:49236)
0923 20:25:22.317119 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:22.317141 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:22.317290 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:22.318708 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:22.318716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:22.318753 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:22.319287 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.319302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.322888 (+ 3586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.322893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.325064 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.325078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.326934 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.326941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.327139 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.327171 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:25:22.327285 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 20:25:22.327419 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":1,"thread_start_us":1368,"threads_started":1}
I20260923 20:25:22.327842 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.316697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.14.127.195:35885)
0923 20:25:22.317958 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.317970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.318008 (+ 38us) server_negotiation.cc:407] Connection header received
0923 20:25:22.318124 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.318134 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.318419 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.319552 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:22.319883 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.321931 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.325256 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.326662 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.326912 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.327108 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.327336 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.327484 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":885,"threads_started":1}
I20260923 20:25:22.330199 30627 tablet_service.cc:1467] Processing CreateTablet for tablet e3271d77f19944209015a5ccfd7fad86 (DEFAULT_TABLE table=test-table [id=d19ccfb002614d5c8dfa8e07df1ba709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:22.331233 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.316649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:35450)
0923 20:25:22.322540 (+ 5891us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:22.322563 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:22.322715 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:22.323405 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:22.323410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:22.323435 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:22.323884 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.323898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.325685 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.325689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.328577 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.328591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.330458 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.330465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.330649 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.330680 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:25:22.330795 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:25:22.330924 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5733,"thread_start_us":5567,"threads_started":1}
I20260923 20:25:22.331323 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.317254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.14.127.193:33351)
0923 20:25:22.322951 (+ 5697us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.322958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.322978 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:25:22.323051 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.323056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.323222 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.323415 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:22.324074 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.325527 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.328766 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.330229 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.330444 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.330595 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.330837 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.330994 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5508,"thread_start_us":5287,"threads_started":1}
I20260923 20:25:22.331477 30760 tablet_service.cc:1467] Processing CreateTablet for tablet e3271d77f19944209015a5ccfd7fad86 (DEFAULT_TABLE table=test-table [id=d19ccfb002614d5c8dfa8e07df1ba709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:22.331930 30627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3271d77f19944209015a5ccfd7fad86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:22.332619 30760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3271d77f19944209015a5ccfd7fad86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:22.336217 30493 tablet_service.cc:1467] Processing CreateTablet for tablet e3271d77f19944209015a5ccfd7fad86 (DEFAULT_TABLE table=test-table [id=d19ccfb002614d5c8dfa8e07df1ba709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:22.337615 30493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3271d77f19944209015a5ccfd7fad86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:22.343883 30843 tablet_bootstrap.cc:492] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Bootstrap starting.
I20260923 20:25:22.346493 30844 tablet_bootstrap.cc:492] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Bootstrap starting.
I20260923 20:25:22.347927 30843 tablet_bootstrap.cc:654] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:22.349366 30843 log.cc:824] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:22.350369 30844 tablet_bootstrap.cc:654] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:22.351086 30846 tablet_bootstrap.cc:492] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Bootstrap starting.
I20260923 20:25:22.351943 30844 log.cc:824] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:22.354457 30846 tablet_bootstrap.cc:654] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:22.354962 30844 tablet_bootstrap.cc:492] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: No bootstrap required, opened a new log
I20260923 20:25:22.355223 30844 ts_tablet_manager.cc:1397] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 20:25:22.355347 30843 tablet_bootstrap.cc:492] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: No bootstrap required, opened a new log
I20260923 20:25:22.355611 30843 ts_tablet_manager.cc:1397] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260923 20:25:22.355773 30846 log.cc:824] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:22.359573 30846 tablet_bootstrap.cc:492] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: No bootstrap required, opened a new log
I20260923 20:25:22.359836 30846 ts_tablet_manager.cc:1397] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260923 20:25:22.363006 30844 raft_consensus.cc:348] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.363396 30844 raft_consensus.cc:374] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:22.363507 30844 raft_consensus.cc:729] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd98534c84c49e985134570c14c8b21, State: Initialized, Role: FOLLOWER
I20260923 20:25:22.364125 30844 consensus_queue.cc:260] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.365782 30825 heartbeater.cc:503] Master 127.14.127.254:32835 was elected leader, sending a full tablet report...
I20260923 20:25:22.365988 30843 raft_consensus.cc:348] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.366416 30843 raft_consensus.cc:374] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:22.366489 30844 ts_tablet_manager.cc:1428] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260923 20:25:22.366501 30843 raft_consensus.cc:729] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6949356cbfb44ef8b0aeea63a43bf49e, State: Initialized, Role: FOLLOWER
I20260923 20:25:22.367142 30843 consensus_queue.cc:260] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.369357 30692 heartbeater.cc:503] Master 127.14.127.254:32835 was elected leader, sending a full tablet report...
I20260923 20:25:22.370003 30843 ts_tablet_manager.cc:1428] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260923 20:25:22.370944 30846 raft_consensus.cc:348] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.371325 30846 raft_consensus.cc:374] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:22.371490 30846 raft_consensus.cc:729] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055e0d5f7fd4c2ebf792773d9b95e15, State: Initialized, Role: FOLLOWER
I20260923 20:25:22.372071 30846 consensus_queue.cc:260] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.373814 30846 ts_tablet_manager.cc:1428] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260923 20:25:22.459616 30849 raft_consensus.cc:482] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:25:22.459949 30849 raft_consensus.cc:504] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.461649 30849 leader_election.cc:288] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6949356cbfb44ef8b0aeea63a43bf49e (127.14.127.194:34589), 9055e0d5f7fd4c2ebf792773d9b95e15 (127.14.127.193:33351)
I20260923 20:25:22.468868 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.462174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:34589 (local address 127.14.127.195:47571)
0923 20:25:22.462382 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:22.462401 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:22.462551 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:22.463242 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:22.463249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:22.463283 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:22.463493 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.463507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.464807 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.464810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.466808 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.466825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.468223 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.468227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.468356 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.468380 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:25:22.468455 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 20:25:22.468546 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 20:25:22.468902 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.462194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:47571 (local address 127.14.127.194:34589)
0923 20:25:22.462392 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.462396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.462475 (+ 79us) server_negotiation.cc:407] Connection header received
0923 20:25:22.462844 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.462848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.462909 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.463039 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:22.463654 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.464670 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.466984 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.468071 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.468227 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.468305 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.468512 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.468663 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:25:22.469558 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.462946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:49873 (local address 127.14.127.193:33351)
0923 20:25:22.463181 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.463187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.463531 (+ 344us) server_negotiation.cc:407] Connection header received
0923 20:25:22.463688 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.463692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.463772 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.463871 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:22.464323 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.465177 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.467371 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.468561 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.468705 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.468775 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.469146 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.469313 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 20:25:22.470075 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.462648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.14.127.195:49873)
0923 20:25:22.463430 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:22.463449 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:22.463568 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:22.463887 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:22.463891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:22.463909 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:22.464165 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.464172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.465337 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.465342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.467220 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:22.467230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.468764 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.468769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:22.468947 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.468982 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:25:22.469749 (+ 767us) client_negotiation.cc:240] Negotiation successful
0923 20:25:22.469855 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":413,"threads_started":1}
I20260923 20:25:22.470018 30647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3271d77f19944209015a5ccfd7fad86" candidate_uuid: "4bd98534c84c49e985134570c14c8b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" is_pre_election: true
I20260923 20:25:22.470585 30647 raft_consensus.cc:2454] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bd98534c84c49e985134570c14c8b21 in term 0.
I20260923 20:25:22.471118 30513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3271d77f19944209015a5ccfd7fad86" candidate_uuid: "4bd98534c84c49e985134570c14c8b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" is_pre_election: true
I20260923 20:25:22.471326 30714 leader_election.cc:302] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bd98534c84c49e985134570c14c8b21, 6949356cbfb44ef8b0aeea63a43bf49e; no voters:
I20260923 20:25:22.471782 30849 raft_consensus.cc:2781] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:25:22.471760 30513 raft_consensus.cc:2454] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bd98534c84c49e985134570c14c8b21 in term 0.
I20260923 20:25:22.471932 30849 raft_consensus.cc:482] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:25:22.472050 30849 raft_consensus.cc:3037] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:22.474249 30849 raft_consensus.cc:504] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.474907 30849 leader_election.cc:288] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [CANDIDATE]: Term 1 election: Requested vote from peers 6949356cbfb44ef8b0aeea63a43bf49e (127.14.127.194:34589), 9055e0d5f7fd4c2ebf792773d9b95e15 (127.14.127.193:33351)
I20260923 20:25:22.475431 30513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3271d77f19944209015a5ccfd7fad86" candidate_uuid: "4bd98534c84c49e985134570c14c8b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15"
I20260923 20:25:22.475538 30647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3271d77f19944209015a5ccfd7fad86" candidate_uuid: "4bd98534c84c49e985134570c14c8b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6949356cbfb44ef8b0aeea63a43bf49e"
I20260923 20:25:22.475654 30513 raft_consensus.cc:3037] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:22.475800 30647 raft_consensus.cc:3037] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:22.478015 30513 raft_consensus.cc:2454] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bd98534c84c49e985134570c14c8b21 in term 1.
I20260923 20:25:22.478209 30647 raft_consensus.cc:2454] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bd98534c84c49e985134570c14c8b21 in term 1.
I20260923 20:25:22.478402 30714 leader_election.cc:302] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bd98534c84c49e985134570c14c8b21, 9055e0d5f7fd4c2ebf792773d9b95e15; no voters:
I20260923 20:25:22.478742 30849 raft_consensus.cc:2781] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:25:22.479199 30849 raft_consensus.cc:686] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd98534c84c49e985134570c14c8b21, State: Running, Role: LEADER
I20260923 20:25:22.479718 30849 consensus_queue.cc:237] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } }
I20260923 20:25:22.484613 30368 catalog_manager.cc:5495] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bd98534c84c49e985134570c14c8b21 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "4bd98534c84c49e985134570c14c8b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } health_report { overall_health: HEALTHY } } }
W20260923 20:25:22.499498 30826 tablet.cc:2369] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:25:22.514740 30693 tablet.cc:2369] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:25:22.521185 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.515634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.14.127.195:35885)
0923 20:25:22.515862 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.515868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.515888 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:25:22.516126 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.516129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.516219 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.516373 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:25:22.516760 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.517487 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.519016 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.519427 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.519567 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.520082 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:25:22.520587 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:25:22.520755 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.520841 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.520925 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260923 20:25:22.522097 30559 tablet.cc:2369] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:25:22.529452 30647 raft_consensus.cc:1261] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 FOLLOWER]: Refusing update from remote peer 4bd98534c84c49e985134570c14c8b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:25:22.529510 30513 raft_consensus.cc:1261] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 1 FOLLOWER]: Refusing update from remote peer 4bd98534c84c49e985134570c14c8b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:25:22.530434 30853 consensus_queue.cc:1035] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:25:22.530872 30849 consensus_queue.cc:1035] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:25:22.540181 30856 mvcc.cc:205] Tried to move back new op lower bound from 7332639221871071232 to 7332639221680943104. Current Snapshot: MvccSnapshot[applied={T|T < 7332639221871071232}]
I20260923 20:25:22.543820 30857 mvcc.cc:205] Tried to move back new op lower bound from 7332639221871071232 to 7332639221680943104. Current Snapshot: MvccSnapshot[applied={T|T < 7332639221871071232}]
I20260923 20:25:22.557308 30367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:25:22.557633 30367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:25:22.561239 30367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Sending DeleteTablet for 3 replicas of tablet e3271d77f19944209015a5ccfd7fad86
I20260923 20:25:22.562423 30627 tablet_service.cc:1514] Processing DeleteTablet for tablet e3271d77f19944209015a5ccfd7fad86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:22 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:38402
I20260923 20:25:22.562713 30493 tablet_service.cc:1514] Processing DeleteTablet for tablet e3271d77f19944209015a5ccfd7fad86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:22 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:35450
I20260923 20:25:22.563470 30865 tablet_replica.cc:330] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: stopping tablet replica
I20260923 20:25:22.563897 30760 tablet_service.cc:1514] Processing DeleteTablet for tablet e3271d77f19944209015a5ccfd7fad86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:22 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:49236
I20260923 20:25:22.563952 30865 raft_consensus.cc:2229] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:25:22.564476 30865 raft_consensus.cc:2258] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:22.566530 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 30430
I20260923 20:25:22.567345 30865 ts_tablet_manager.cc:1905] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:25:22.576586 30865 ts_tablet_manager.cc:1918] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:25:22.576895 30865 log.cc:1196] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/e3271d77f19944209015a5ccfd7fad86
I20260923 20:25:22.577653 30865 ts_tablet_manager.cc:1939] T e3271d77f19944209015a5ccfd7fad86 P 6949356cbfb44ef8b0aeea63a43bf49e: Deleting consensus metadata
I20260923 20:25:22.580147 30353 catalog_manager.cc:4841] TS 6949356cbfb44ef8b0aeea63a43bf49e (127.14.127.194:34589): tablet e3271d77f19944209015a5ccfd7fad86 (table test-table [id=d19ccfb002614d5c8dfa8e07df1ba709]) successfully deleted
I20260923 20:25:22.584025 30867 tablet_replica.cc:330] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: stopping tablet replica
I20260923 20:25:22.584641 30867 raft_consensus.cc:2229] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:25:22.585450 30867 raft_consensus.cc:2258] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:22.588083 30867 ts_tablet_manager.cc:1905] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:25:22.591099 30866 tablet_replica.cc:330] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: stopping tablet replica
I20260923 20:25:22.591565 30866 raft_consensus.cc:2229] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:25:22.592061 30866 raft_consensus.cc:2258] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:22.594220 30866 ts_tablet_manager.cc:1905] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:25:22.596371 30867 ts_tablet_manager.cc:1918] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:25:22.596604 30867 log.cc:1196] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/e3271d77f19944209015a5ccfd7fad86
I20260923 20:25:22.597230 30867 ts_tablet_manager.cc:1939] T e3271d77f19944209015a5ccfd7fad86 P 4bd98534c84c49e985134570c14c8b21: Deleting consensus metadata
Sep 23 20:25:22 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:25:22.599332 30355 catalog_manager.cc:4841] TS 4bd98534c84c49e985134570c14c8b21 (127.14.127.195:35885): tablet e3271d77f19944209015a5ccfd7fad86 (table test-table [id=d19ccfb002614d5c8dfa8e07df1ba709]) successfully deleted
I20260923 20:25:22.601243 30866 ts_tablet_manager.cc:1918] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:25:22.601444 30866 log.cc:1196] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/e3271d77f19944209015a5ccfd7fad86
I20260923 20:25:22.602080 30866 ts_tablet_manager.cc:1939] T e3271d77f19944209015a5ccfd7fad86 P 9055e0d5f7fd4c2ebf792773d9b95e15: Deleting consensus metadata
I20260923 20:25:22.603742 30353 catalog_manager.cc:4841] TS 9055e0d5f7fd4c2ebf792773d9b95e15 (127.14.127.193:33351): tablet e3271d77f19944209015a5ccfd7fad86 (table test-table [id=d19ccfb002614d5c8dfa8e07df1ba709]) successfully deleted
I20260923 20:25:22.606832 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.567791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.14.127.193:33351)
0923 20:25:22.567985 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.567991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.583304 (+ 15313us) server_negotiation.cc:407] Connection header received
0923 20:25:22.583526 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.583534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.583637 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.583750 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:22.586088 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.587368 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.589375 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.590057 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.590156 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.599578 (+ 9422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:22.599599 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:22.599623 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:22.599668 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:22.603795 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.604607 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.604617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.604621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.604792 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.605185 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.605192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.605196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.605655 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:22.605974 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.606184 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.606413 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:25:22.607631 30553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:22.714018 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 30430
I20260923 20:25:22.739975 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 30564
Sep 23 20:25:22 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:25:22.757364 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.740756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.14.127.194:34589)
0923 20:25:22.740937 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.740941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.741013 (+ 72us) server_negotiation.cc:407] Connection header received
0923 20:25:22.741203 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.741207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.741278 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.741354 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:22.743118 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.743853 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.745701 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.746059 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.746148 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.751613 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:22.751627 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:22.751642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:22.751677 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:22.754440 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.755188 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.755198 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.755200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.755330 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.755737 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.755741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.755743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.756107 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:22.756424 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.756801 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.757030 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:25:22.757999 30687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:22.848816 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 30564
I20260923 20:25:22.866128 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 30697
Sep 23 20:25:22 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:25:22.884052 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:22.866999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.14.127.195:35885)
0923 20:25:22.867233 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:22.867240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:22.867328 (+ 88us) server_negotiation.cc:407] Connection header received
0923 20:25:22.867486 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:22.867492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:22.867582 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:22.867718 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:22.868936 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.870091 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.871511 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:22.872204 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:22.872337 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:22.876853 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:22.876870 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:22.876974 (+ 104us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:22.877025 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:22.881114 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.881779 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.881789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.881794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.882015 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:22.882373 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:22.882379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:22.882382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:22.882892 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:22.883217 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:22.883394 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 20:25:22.883608 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260923 20:25:22.884905 30820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:23.006325 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 30697
I20260923 20:25:23.026190 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 30336
I20260923 20:25:23.027333 30398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:23.137588 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 30336
I20260923 20:25:43.159055 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:32835
--webserver_interface=127.14.127.254
--webserver_port=35547
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:25:43.491200 30873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:43.491647 30873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:43.491815 30873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:43.499323 30873 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:25:43.499467 30873 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:25:43.499558 30873 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:25:43.499632 30873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:25:43.499732 30873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:43.499832 30873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:43.499922 30873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:25:43.500000 30873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:25:43.510293 30873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=35547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:43.511739 30873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:43.513096 30873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:43.538488 30879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:43.538951 30873 server_base.cc:1034] running on GCE node
W20260923 20:25:43.538527 30882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:43.538578 30880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:43.540225 30873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:43.541944 30873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:43.543249 30873 hybrid_clock.cc:648] HybridClock initialized: now 1790195143543166 us; error 53 us; skew 500 ppm
I20260923 20:25:43.543666 30873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:43 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195143, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:43.547528 30873 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:25:43.549247 30873 webserver.cc:460] Webserver started at http://127.14.127.254:35547/ using document root <none> and password file <none>
I20260923 20:25:43.550032 30873 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:43.550145 30873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:43.556324 30873 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:25:43.559479 30889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:43.561074 30873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:25:43.561272 30873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "8a4a0a2999b44936ba4d9b04f37f9eec"
format_stamp: "Formatted at 2026-09-23 20:25:20 on dist-test-slave-wmd0"
server_key: "ee42adf781353a2627a4668e08683a25"
server_key_iv: "9c40da1e8c197da3fdeb85426833b893"
server_key_version: "encryptionkey@0"
I20260923 20:25:43.562474 30873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:43.580297 30873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:43.585412 30873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:43.585682 30873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:43.605844 30873 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:32835
I20260923 20:25:43.605867 30940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:32835 every 8 connection(s)
I20260923 20:25:43.607605 30873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:25:43.613066 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 30873
I20260923 20:25:43.613998 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:33351
--local_ip_for_outbound_sockets=127.14.127.193
--tserver_master_addrs=127.14.127.254:32835
--webserver_port=36609
--webserver_interface=127.14.127.193
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:25:43.615048 30941 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:25:43.618350 30941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Bootstrap starting.
I20260923 20:25:43.646188 30941 log.cc:824] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:43.661813 30941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:25:43.662472 30941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Bootstrap complete.
I20260923 20:25:43.676970 30941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:25:43.678201 30941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4a0a2999b44936ba4d9b04f37f9eec, State: Initialized, Role: FOLLOWER
I20260923 20:25:43.678968 30941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:25:43.679255 30941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:25:43.679388 30941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:25:43.679574 30941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:25:43.686390 30941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:25:43.687122 30941 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a4a0a2999b44936ba4d9b04f37f9eec; no voters:
I20260923 20:25:43.687855 30941 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:25:43.688033 30945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:25:43.689435 30945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 2 LEADER]: Becoming Leader. State: Replica: 8a4a0a2999b44936ba4d9b04f37f9eec, State: Running, Role: LEADER
I20260923 20:25:43.690112 30941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:25:43.690037 30945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:25:43.693179 30947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4a0a2999b44936ba4d9b04f37f9eec. Latest consensus state: current_term: 2 leader_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } } }
I20260923 20:25:43.699581 30947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: This master's current role is: LEADER
I20260923 20:25:43.693276 30946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } } }
I20260923 20:25:43.699923 30946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: This master's current role is: LEADER
I20260923 20:25:43.716581 30961 catalog_manager.cc:1259] Loaded cluster ID: eaef0f26cbd5438790cd497dff794cff
I20260923 20:25:43.716717 30961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: loading cluster ID for follower catalog manager: success
I20260923 20:25:43.720945 30961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: acquiring CA information for follower catalog manager: success
I20260923 20:25:43.723932 30961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:25:43.724471 30953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:25:43.729739 30953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:25:43.731030 30953 catalog_manager.cc:1259] Loaded cluster ID: eaef0f26cbd5438790cd497dff794cff
I20260923 20:25:43.731128 30953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:25:43.732841 30953 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:25:43.734068 30953 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Loaded TSK: 0
I20260923 20:25:43.770033 30953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Generated new TSK 1
I20260923 20:25:43.770438 30953 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:25:43.996914 30943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:43.997267 30943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:43.997450 30943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:44.005002 30943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:44.005158 30943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:44.005352 30943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:25:44.016623 30943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:33351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=36609
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:44.018483 30943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:44.020192 30943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:44.036332 30968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:44.038188 30969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:44.039315 30971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:44.039644 30943 server_base.cc:1034] running on GCE node
I20260923 20:25:44.040416 30943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:44.041687 30943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:44.042927 30943 hybrid_clock.cc:648] HybridClock initialized: now 1790195144042874 us; error 52 us; skew 500 ppm
I20260923 20:25:44.043324 30943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:44 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195144, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:44.047371 30943 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:25:44.049254 30943 webserver.cc:460] Webserver started at http://127.14.127.193:36609/ using document root <none> and password file <none>
I20260923 20:25:44.050192 30943 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:44.050349 30943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:44.057451 30943 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:25:44.060755 30978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:44.062271 30943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:25:44.062454 30943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "9055e0d5f7fd4c2ebf792773d9b95e15"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "ad4bb5c5c5309bbdfde13e821dde93b7"
server_key_iv: "11f847fd3465290f8d937728ad96e70d"
server_key_version: "encryptionkey@0"
I20260923 20:25:44.063498 30943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:44.090325 30943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:44.094553 30943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:44.094852 30943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:44.095810 30943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:44.097043 30943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:44.097131 30943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:44.097213 30943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:44.097270 30943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:44.129858 30943 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:33351
I20260923 20:25:44.129964 31090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:33351 every 8 connection(s)
I20260923 20:25:44.131419 30943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:25:44.133090 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 30943
I20260923 20:25:44.133724 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:34589
--local_ip_for_outbound_sockets=127.14.127.194
--tserver_master_addrs=127.14.127.254:32835
--webserver_port=43273
--webserver_interface=127.14.127.194
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:25:44 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195144, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:44.158577 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:44.134940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:39379)
0923 20:25:44.135855 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:44.135901 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:44.138586 (+ 2685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:44.140268 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:44.140280 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:44.141042 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:44.141716 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:44.141734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:44.144198 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:44.144215 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:44.145370 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:44.145381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:44.146875 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:44.146881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:44.147076 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:44.147833 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:44.147867 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:44.150343 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:44.154216 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:44.154227 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:44.154242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:44.154739 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:44.155183 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:44.155187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:44.155191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:44.155375 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:44.156122 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:44.156130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:44.156458 (+ 328us) client_negotiation.cc:769] Sending connection context
0923 20:25:44.156734 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 20:25:44.157279 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":378,"threads_started":1}
I20260923 20:25:44.158576 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:44.134752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:39379 (local address 127.14.127.254:32835)
0923 20:25:44.135458 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:44.135475 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:44.136058 (+ 583us) server_negotiation.cc:407] Connection header received
0923 20:25:44.139238 (+ 3180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:44.139259 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:44.139729 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:44.140133 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:44.141900 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:44.144026 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:44.145535 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:44.146668 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:44.146865 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:44.150687 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:44.150702 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:44.150726 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:44.150778 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:44.154059 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:44.154878 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:44.154885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:44.154891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:44.155055 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:44.155488 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:44.155494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:44.155497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:44.155970 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:44.156177 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:44.156956 (+ 779us) server_negotiation.cc:299] Negotiation successful
0923 20:25:44.157367 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":285,"threads_started":1}
I20260923 20:25:44.161386 31091 heartbeater.cc:347] Connected to a master server at 127.14.127.254:32835
I20260923 20:25:44.161700 31091 heartbeater.cc:464] Registering TS with master...
I20260923 20:25:44.162398 31091 heartbeater.cc:511] Master 127.14.127.254:32835 requested a full tablet report, sending...
I20260923 20:25:44.164541 30906 ts_manager.cc:194] Registered new tserver with Master: 9055e0d5f7fd4c2ebf792773d9b95e15 (127.14.127.193:33351)
I20260923 20:25:44.166124 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:39379
W20260923 20:25:44.463627 31095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:44.463975 31095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:44.464148 31095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:44.471774 31095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:44.471930 31095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:44.472117 31095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:25:44.482053 31095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:34589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=43273
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:44.483415 31095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:44.484748 31095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:44.498072 31100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:44.498039 31101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:44.498106 31103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:44.498881 31095 server_base.cc:1034] running on GCE node
I20260923 20:25:44.499519 31095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:44.500483 31095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:44.501724 31095 hybrid_clock.cc:648] HybridClock initialized: now 1790195144501682 us; error 36 us; skew 500 ppm
I20260923 20:25:44.502075 31095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:44 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195144, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:44.505312 31095 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:25:44.506910 31095 webserver.cc:460] Webserver started at http://127.14.127.194:43273/ using document root <none> and password file <none>
I20260923 20:25:44.507586 31095 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:44.507711 31095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:44.513072 31095 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 20:25:44.515764 31110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:44.517118 31095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:25:44.517254 31095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "6949356cbfb44ef8b0aeea63a43bf49e"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "69f8f3b786b250ac60c7dff87f136b7a"
server_key_iv: "25b14fb9333d59abc423d5cd0e84b6ab"
server_key_version: "encryptionkey@0"
I20260923 20:25:44.518177 31095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:44.545708 31095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:44.550514 31095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:44.550829 31095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:44.551831 31095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:44.553169 31095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:44.553275 31095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:44.553350 31095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:44.553406 31095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:44.585481 31095 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:34589
I20260923 20:25:44.585572 31222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:34589 every 8 connection(s)
I20260923 20:25:44.587255 31095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:25:44.590255 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 31095
I20260923 20:25:44.590981 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:35885
--local_ip_for_outbound_sockets=127.14.127.195
--tserver_master_addrs=127.14.127.254:32835
--webserver_port=40377
--webserver_interface=127.14.127.195
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:25:44 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195144, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:44.612630 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:44.589916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:40879 (local address 127.14.127.254:32835)
0923 20:25:44.590317 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:44.590324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:44.591768 (+ 1444us) server_negotiation.cc:407] Connection header received
0923 20:25:44.594756 (+ 2988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:44.594761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:44.594847 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:44.594938 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:44.597170 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:44.597876 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:44.600084 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:44.600864 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:44.601024 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:44.604525 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:44.604535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:44.604544 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:44.604592 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:44.608616 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:44.609648 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:44.609657 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:44.609663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:44.609818 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:44.610311 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:44.610319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:44.610325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:44.610803 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:44.610973 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:44.611852 (+ 879us) server_negotiation.cc:299] Negotiation successful
0923 20:25:44.612152 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260923 20:25:44.613927 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:44.590598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:40879)
0923 20:25:44.591549 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:44.591600 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:44.594347 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:44.595226 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:44.595242 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:44.596083 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:44.596965 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:44.596995 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:44.598042 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:44.598050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:44.599877 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:44.599890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:44.601082 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:44.601086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:44.601279 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:44.601973 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:44.601996 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:44.604266 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:44.608805 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:44.608818 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:44.608835 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:44.609509 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:44.609934 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:44.609939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:44.609943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:44.610170 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:44.610962 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:44.610971 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:44.611384 (+ 413us) client_negotiation.cc:769] Sending connection context
0923 20:25:44.611706 (+ 322us) client_negotiation.cc:240] Negotiation successful
0923 20:25:44.612233 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":377,"threads_started":1}
I20260923 20:25:44.615564 31223 heartbeater.cc:347] Connected to a master server at 127.14.127.254:32835
I20260923 20:25:44.615937 31223 heartbeater.cc:464] Registering TS with master...
I20260923 20:25:44.616889 31223 heartbeater.cc:511] Master 127.14.127.254:32835 requested a full tablet report, sending...
I20260923 20:25:44.618702 30906 ts_manager.cc:194] Registered new tserver with Master: 6949356cbfb44ef8b0aeea63a43bf49e (127.14.127.194:34589)
I20260923 20:25:44.619853 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:40879
W20260923 20:25:44.929332 31226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:25:44.929679 31226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:25:44.929855 31226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:25:44.937425 31226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:25:44.937589 31226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:25:44.937763 31226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:25:44.947795 31226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:35885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=40377
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:25:44.949188 31226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:25:44.950522 31226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:25:44.963960 31232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:44.963967 31231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:25:44.964432 31234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:25:44.968215 31226 server_base.cc:1034] running on GCE node
I20260923 20:25:44.968878 31226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:25:44.969822 31226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:25:44.971045 31226 hybrid_clock.cc:648] HybridClock initialized: now 1790195144970983 us; error 59 us; skew 500 ppm
I20260923 20:25:44.971360 31226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:25:44 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195144, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:25:44.974399 31226 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:25:44.975970 31226 webserver.cc:460] Webserver started at http://127.14.127.195:40377/ using document root <none> and password file <none>
I20260923 20:25:44.976749 31226 fs_manager.cc:359] Metadata directory not provided
I20260923 20:25:44.976905 31226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:25:44.982203 31226 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:25:44.984892 31241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:44.986213 31226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:25:44.986389 31226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "4bd98534c84c49e985134570c14c8b21"
format_stamp: "Formatted at 2026-09-23 20:25:22 on dist-test-slave-wmd0"
server_key: "24fdf737ac3b233fcf00aa46d3c400a9"
server_key_iv: "1acce4ce8024d3f19fd0f33962639a86"
server_key_version: "encryptionkey@0"
I20260923 20:25:44.987442 31226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:25:45.009204 31226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:25:45.013497 31226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:25:45.013734 31226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:25:45.014820 31226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:25:45.016220 31226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:25:45.016309 31226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:45.016392 31226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:25:45.016446 31226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:25:45.048682 31226 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:35885
I20260923 20:25:45.048766 31353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:35885 every 8 connection(s)
I20260923 20:25:45.052336 31226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:25:45.053397 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 31226
Sep 23 20:25:45 dist-test-slave-wmd0 krb5kdc[30323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195144, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:25:45.076275 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.054503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.14.127.254:32835)
0923 20:25:45.054752 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.054758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.055204 (+ 446us) server_negotiation.cc:407] Connection header received
0923 20:25:45.055415 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.055422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.055505 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.055617 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:45.057941 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.058751 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.060168 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.060584 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.060733 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.069993 (+ 9260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:45.069999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:45.070002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:45.070035 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:45.073306 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.073976 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.073984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.073987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.074129 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.074501 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.074507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.074511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.074950 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:45.075338 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.075658 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.075889 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":76}
I20260923 20:25:45.076887 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.055333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:47995 (local address 127.14.127.254:32835)
0923 20:25:45.057490 (+ 2157us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.057496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.057515 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:25:45.059021 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.059026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.059103 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.059227 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:45.060708 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.061596 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.063240 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.063665 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.063802 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.070365 (+ 6563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:45.070370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:45.070373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:45.070412 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:45.073365 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.074090 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.074097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.074105 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.074220 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.074704 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.074709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.074712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.075050 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:45.075234 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.076196 (+ 962us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.076385 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":2043,"thread_start_us":1900,"threads_started":1}
I20260923 20:25:45.078223 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.055951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:47995)
0923 20:25:45.056771 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.056817 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:45.058765 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:45.059470 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:45.059478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:45.059976 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:25:45.060524 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.060539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.061773 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.061777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.063085 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.063097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.063920 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.063928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.064149 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.067897 (+ 3748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:25:45.067926 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:25:45.070102 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:25:45.073506 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:45.073516 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:45.073533 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:45.073957 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:45.074356 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:25:45.074361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:25:45.074364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:25:45.074553 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:25:45.075199 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:25:45.075207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:25:45.075673 (+ 466us) client_negotiation.cc:769] Sending connection context
0923 20:25:45.075982 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 20:25:45.076550 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":353,"threads_started":1}
I20260923 20:25:45.079631 31354 heartbeater.cc:347] Connected to a master server at 127.14.127.254:32835
I20260923 20:25:45.079895 31354 heartbeater.cc:464] Registering TS with master...
I20260923 20:25:45.080542 31354 heartbeater.cc:511] Master 127.14.127.254:32835 requested a full tablet report, sending...
I20260923 20:25:45.081836 30906 ts_manager.cc:194] Registered new tserver with Master: 4bd98534c84c49e985134570c14c8b21 (127.14.127.195:35885)
I20260923 20:25:45.082579 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:47995
I20260923 20:25:45.091364 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:25:45.097556 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.092339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.14.127.254:32835)
0923 20:25:45.092571 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.092577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.092925 (+ 348us) server_negotiation.cc:407] Connection header received
0923 20:25:45.093115 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.093119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.093240 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.093358 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:25:45.093714 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.094397 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.095827 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.096222 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.096315 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.096767 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:25:45.097045 (+ 278us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 20:25:45.097305 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34868: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 20:25:45.097697 31357 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34868: token expired
W20260923 20:25:45.098212 30830 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.127.254:32835)
I20260923 20:25:45.111199 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.098670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.14.127.254:32835)
0923 20:25:45.098827 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.098831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.098845 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:25:45.098883 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.098886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.098937 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.099010 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:45.100838 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.101514 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.102982 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.103383 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.103478 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.106025 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:45.106029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:45.106032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:45.106058 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:45.108717 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.109432 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.109436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.109438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.109562 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.109958 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.109961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.109963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.110266 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:45.110459 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.110773 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.110933 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:25:45.114406 30906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:25:45.121907 30906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:25:45.147347 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.133004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:54162)
0923 20:25:45.137565 (+ 4561us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.137621 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:45.137832 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:45.139798 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:45.139806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:45.139837 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:45.140233 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.140247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.142300 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.142305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.144857 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.144878 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.146262 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.146272 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.146478 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.146564 (+ 86us) client_negotiation.cc:769] Sending connection context
0923 20:25:45.146731 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 20:25:45.146933 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4231,"mutex_wait_us":5,"thread_start_us":4036,"threads_started":1}
I20260923 20:25:45.147321 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.133276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.14.127.193:33351)
0923 20:25:45.139295 (+ 6019us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.139301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.139324 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:25:45.139396 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.139402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.139581 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.139778 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:45.140473 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.142127 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.145046 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.145986 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.146242 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.146426 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.146793 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.146977 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5856,"thread_start_us":5717,"threads_started":1}
I20260923 20:25:45.149438 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.139587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:34589 (local address 127.0.0.1:34270)
0923 20:25:45.140721 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.140746 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:45.140938 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:45.141719 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:45.141742 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:45.141767 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:45.142099 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.142107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.143585 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.143590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.146347 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.146362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.148593 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.148599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.148771 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.148801 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:25:45.148903 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 20:25:45.149037 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":47,"thread_start_us":868,"threads_started":1}
I20260923 20:25:45.149497 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.138308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.14.127.194:34589)
0923 20:25:45.140394 (+ 2086us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.140404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.140838 (+ 434us) server_negotiation.cc:407] Connection header received
0923 20:25:45.141141 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.141149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.141401 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.141563 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:45.142267 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.143408 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.146593 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.148112 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.148292 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.148447 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.148968 (+ 521us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.149163 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1911,"thread_start_us":1790,"threads_started":1}
I20260923 20:25:45.149984 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.133284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:35885 (local address 127.0.0.1:58922)
0923 20:25:45.138532 (+ 5248us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.138554 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:45.138716 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:45.141473 (+ 2757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:45.141478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:45.141504 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:45.141891 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.141904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.144209 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.144214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.146574 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.146594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.149263 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.149272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.149453 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.149496 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 20:25:45.149566 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 20:25:45.149694 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5097,"mutex_wait_us":55,"thread_start_us":4927,"threads_started":1}
I20260923 20:25:45.151015 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.133948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.14.127.195:35885)
0923 20:25:45.140895 (+ 6947us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.140904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.140936 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:25:45.141025 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.141033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.141269 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.142078 (+ 809us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:45.142398 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.144017 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.146822 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.149014 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.150281 (+ 1267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.150397 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.150447 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.150651 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6762,"thread_start_us":6835,"threads_started":1}
I20260923 20:25:45.152052 31026 tablet_service.cc:1467] Processing CreateTablet for tablet c68672ad3cb145e7aad58ef8bf51fd61 (DEFAULT_TABLE table=test-table [id=15602de8cda1488cbf1e9ee15bb9ae00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:45.153090 31158 tablet_service.cc:1467] Processing CreateTablet for tablet c68672ad3cb145e7aad58ef8bf51fd61 (DEFAULT_TABLE table=test-table [id=15602de8cda1488cbf1e9ee15bb9ae00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:45.153661 31026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68672ad3cb145e7aad58ef8bf51fd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:45.154199 31158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68672ad3cb145e7aad58ef8bf51fd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:45.154495 31289 tablet_service.cc:1467] Processing CreateTablet for tablet c68672ad3cb145e7aad58ef8bf51fd61 (DEFAULT_TABLE table=test-table [id=15602de8cda1488cbf1e9ee15bb9ae00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:25:45.155843 31289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68672ad3cb145e7aad58ef8bf51fd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:25:45.166051 31370 tablet_bootstrap.cc:492] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: Bootstrap starting.
I20260923 20:25:45.169344 31371 tablet_bootstrap.cc:492] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Bootstrap starting.
I20260923 20:25:45.169546 31091 heartbeater.cc:503] Master 127.14.127.254:32835 was elected leader, sending a full tablet report...
I20260923 20:25:45.171020 31372 tablet_bootstrap.cc:492] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: Bootstrap starting.
I20260923 20:25:45.172982 31371 tablet_bootstrap.cc:654] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:45.173662 31372 tablet_bootstrap.cc:654] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:45.174324 31371 log.cc:824] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:45.174813 31372 log.cc:824] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:45.175076 31370 tablet_bootstrap.cc:654] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: Neither blocks nor log segments found. Creating new log.
I20260923 20:25:45.176357 31370 log.cc:824] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:25:45.177595 31371 tablet_bootstrap.cc:492] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: No bootstrap required, opened a new log
I20260923 20:25:45.177874 31371 ts_tablet_manager.cc:1397] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260923 20:25:45.178004 31372 tablet_bootstrap.cc:492] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: No bootstrap required, opened a new log
I20260923 20:25:45.178269 31372 ts_tablet_manager.cc:1397] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260923 20:25:45.179816 31370 tablet_bootstrap.cc:492] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: No bootstrap required, opened a new log
I20260923 20:25:45.180063 31370 ts_tablet_manager.cc:1397] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260923 20:25:45.186590 31370 raft_consensus.cc:348] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.187270 31370 raft_consensus.cc:374] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:45.187393 31370 raft_consensus.cc:729] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6949356cbfb44ef8b0aeea63a43bf49e, State: Initialized, Role: FOLLOWER
I20260923 20:25:45.188194 31370 consensus_queue.cc:260] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.188773 31371 raft_consensus.cc:348] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.188771 31372 raft_consensus.cc:348] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.189195 31371 raft_consensus.cc:374] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:45.189206 31372 raft_consensus.cc:374] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:25:45.189303 31372 raft_consensus.cc:729] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd98534c84c49e985134570c14c8b21, State: Initialized, Role: FOLLOWER
I20260923 20:25:45.189303 31371 raft_consensus.cc:729] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9055e0d5f7fd4c2ebf792773d9b95e15, State: Initialized, Role: FOLLOWER
I20260923 20:25:45.189538 31223 heartbeater.cc:503] Master 127.14.127.254:32835 was elected leader, sending a full tablet report...
I20260923 20:25:45.189873 31371 consensus_queue.cc:260] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.189899 31372 consensus_queue.cc:260] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.190212 31370 ts_tablet_manager.cc:1428] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260923 20:25:45.191640 31354 heartbeater.cc:503] Master 127.14.127.254:32835 was elected leader, sending a full tablet report...
I20260923 20:25:45.191833 31371 ts_tablet_manager.cc:1428] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 20:25:45.191939 31372 ts_tablet_manager.cc:1428] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260923 20:25:45.196305 31376 raft_consensus.cc:482] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:25:45.196556 31376 raft_consensus.cc:504] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.198187 31376 leader_election.cc:288] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9055e0d5f7fd4c2ebf792773d9b95e15 (127.14.127.193:33351), 4bd98534c84c49e985134570c14c8b21 (127.14.127.195:35885)
I20260923 20:25:45.206116 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.199064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:38733 (local address 127.14.127.193:33351)
0923 20:25:45.199277 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.199284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.199596 (+ 312us) server_negotiation.cc:407] Connection header received
0923 20:25:45.199742 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.199750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.199838 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.199938 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:45.200537 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.201542 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.204109 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.205277 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.205436 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.205515 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.205796 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.205889 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:25:45.206406 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.198534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.14.127.194:38733)
0923 20:25:45.199491 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.199507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:45.199629 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:45.199980 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:45.199985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:45.200015 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:45.200405 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.200418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.201675 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.201679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.203963 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.203978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.205470 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.205475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.205576 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.205642 (+ 66us) client_negotiation.cc:769] Sending connection context
0923 20:25:45.205752 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 20:25:45.205890 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":69,"thread_start_us":1287,"threads_started":1}
I20260923 20:25:45.206524 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.198737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:35885 (local address 127.14.127.194:58421)
0923 20:25:45.200147 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.200169 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:25:45.200293 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:25:45.200608 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:25:45.200611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:25:45.200631 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:25:45.200910 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.200916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.202086 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.202124 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.204190 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:25:45.204200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.205733 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.205742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:25:45.205903 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.205931 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 20:25:45.206021 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 20:25:45.206138 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"mutex_wait_us":2,"thread_start_us":468,"threads_started":1}
I20260923 20:25:45.206547 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.199260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:58421 (local address 127.14.127.195:35885)
0923 20:25:45.199419 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.199423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.200259 (+ 836us) server_negotiation.cc:407] Connection header received
0923 20:25:45.200414 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.200417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.200493 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.200588 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:25:45.201051 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.201952 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.204313 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.205524 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.205649 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.205695 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.206055 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.206191 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:25:45.207641 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68672ad3cb145e7aad58ef8bf51fd61" candidate_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bd98534c84c49e985134570c14c8b21" is_pre_election: true
I20260923 20:25:45.207653 31046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68672ad3cb145e7aad58ef8bf51fd61" candidate_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" is_pre_election: true
I20260923 20:25:45.208196 31046 raft_consensus.cc:2454] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6949356cbfb44ef8b0aeea63a43bf49e in term 0.
I20260923 20:25:45.208196 31309 raft_consensus.cc:2454] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6949356cbfb44ef8b0aeea63a43bf49e in term 0.
I20260923 20:25:45.208884 31114 leader_election.cc:302] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bd98534c84c49e985134570c14c8b21, 6949356cbfb44ef8b0aeea63a43bf49e; no voters:
I20260923 20:25:45.209231 31376 raft_consensus.cc:2781] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:25:45.209383 31376 raft_consensus.cc:482] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:25:45.209510 31376 raft_consensus.cc:3037] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:45.212189 31376 raft_consensus.cc:504] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.212762 31376 leader_election.cc:288] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [CANDIDATE]: Term 1 election: Requested vote from peers 9055e0d5f7fd4c2ebf792773d9b95e15 (127.14.127.193:33351), 4bd98534c84c49e985134570c14c8b21 (127.14.127.195:35885)
I20260923 20:25:45.213233 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68672ad3cb145e7aad58ef8bf51fd61" candidate_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bd98534c84c49e985134570c14c8b21"
I20260923 20:25:45.213233 31046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68672ad3cb145e7aad58ef8bf51fd61" candidate_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15"
I20260923 20:25:45.213435 31309 raft_consensus.cc:3037] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:45.213435 31046 raft_consensus.cc:3037] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:25:45.216614 31309 raft_consensus.cc:2454] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6949356cbfb44ef8b0aeea63a43bf49e in term 1.
I20260923 20:25:45.216622 31046 raft_consensus.cc:2454] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6949356cbfb44ef8b0aeea63a43bf49e in term 1.
I20260923 20:25:45.217056 31114 leader_election.cc:302] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bd98534c84c49e985134570c14c8b21, 6949356cbfb44ef8b0aeea63a43bf49e; no voters:
I20260923 20:25:45.217347 31376 raft_consensus.cc:2781] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:25:45.217782 31376 raft_consensus.cc:686] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 LEADER]: Becoming Leader. State: Replica: 6949356cbfb44ef8b0aeea63a43bf49e, State: Running, Role: LEADER
I20260923 20:25:45.218251 31376 consensus_queue.cc:237] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } }
I20260923 20:25:45.223064 30905 catalog_manager.cc:5495] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6949356cbfb44ef8b0aeea63a43bf49e (127.14.127.194). New cstate: current_term: 1 leader_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6949356cbfb44ef8b0aeea63a43bf49e" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34589 } health_report { overall_health: HEALTHY } } }
I20260923 20:25:45.239164 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.234155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.14.127.194:34589)
0923 20:25:45.234392 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.234397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.234414 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:25:45.234598 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.234603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.234731 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.234849 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:25:45.235185 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.235863 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.237110 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.237488 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.237575 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.238006 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:25:45.238506 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:25:45.238662 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.238789 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.238850 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 20:25:45.246824 31046 raft_consensus.cc:1261] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 1 FOLLOWER]: Refusing update from remote peer 6949356cbfb44ef8b0aeea63a43bf49e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:25:45.246825 31309 raft_consensus.cc:1261] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 1 FOLLOWER]: Refusing update from remote peer 6949356cbfb44ef8b0aeea63a43bf49e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:25:45.247771 31381 consensus_queue.cc:1035] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9055e0d5f7fd4c2ebf792773d9b95e15" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:25:45.248086 31376 consensus_queue.cc:1035] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bd98534c84c49e985134570c14c8b21" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 35885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:25:45.259624 31385 mvcc.cc:205] Tried to move back new op lower bound from 7332639314921730048 to 7332639314817875968. Current Snapshot: MvccSnapshot[applied={T|T < 7332639314921730048}]
I20260923 20:25:45.266713 31386 mvcc.cc:205] Tried to move back new op lower bound from 7332639314921730048 to 7332639314817875968. Current Snapshot: MvccSnapshot[applied={T|T < 7332639314921730048}]
I20260923 20:25:45.285436 30906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:25:45.285763 30906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:25:45.289484 30906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Sending DeleteTablet for 3 replicas of tablet c68672ad3cb145e7aad58ef8bf51fd61
I20260923 20:25:45.290879 31289 tablet_service.cc:1514] Processing DeleteTablet for tablet c68672ad3cb145e7aad58ef8bf51fd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:45 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:58922
I20260923 20:25:45.291016 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 30943
I20260923 20:25:45.291307 31026 tablet_service.cc:1514] Processing DeleteTablet for tablet c68672ad3cb145e7aad58ef8bf51fd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:45 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:54162
I20260923 20:25:45.292028 31158 tablet_service.cc:1514] Processing DeleteTablet for tablet c68672ad3cb145e7aad58ef8bf51fd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:25:45 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:34270
I20260923 20:25:45.303028 31395 tablet_replica.cc:330] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: stopping tablet replica
I20260923 20:25:45.305565 31397 tablet_replica.cc:330] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: stopping tablet replica
I20260923 20:25:45.306284 31397 raft_consensus.cc:2229] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:25:45.307199 31397 raft_consensus.cc:2258] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:45.307235 31395 raft_consensus.cc:2229] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:25:45.308120 31396 tablet_replica.cc:330] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: stopping tablet replica
I20260923 20:25:45.308187 31395 raft_consensus.cc:2258] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:45.308641 31396 raft_consensus.cc:2229] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:25:45.309217 31396 raft_consensus.cc:2258] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:25:45.310104 31397 ts_tablet_manager.cc:1905] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:25:45.310652 31395 ts_tablet_manager.cc:1905] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:25:45.311581 31396 ts_tablet_manager.cc:1905] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:25:45.314765 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.292528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.14.127.193:33351)
0923 20:25:45.292781 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.292790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.292814 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:25:45.292887 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.292896 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.293020 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.293149 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:45.296301 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.297871 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.300556 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.301470 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.301636 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.306862 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:45.306882 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:45.306900 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:45.306948 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:45.311330 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.312284 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.312294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.312299 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.312465 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.312939 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.312944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.312947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.313438 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:45.313754 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.314109 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.314351 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:25:45.315796 31086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:45.319962 31395 ts_tablet_manager.cc:1918] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:25:45.320147 31395 log.cc:1196] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/c68672ad3cb145e7aad58ef8bf51fd61
I20260923 20:25:45.320729 31395 ts_tablet_manager.cc:1939] T c68672ad3cb145e7aad58ef8bf51fd61 P 4bd98534c84c49e985134570c14c8b21: Deleting consensus metadata
I20260923 20:25:45.320977 31397 ts_tablet_manager.cc:1918] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:25:45.321254 31397 log.cc:1196] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/c68672ad3cb145e7aad58ef8bf51fd61
I20260923 20:25:45.321902 31397 ts_tablet_manager.cc:1939] T c68672ad3cb145e7aad58ef8bf51fd61 P 6949356cbfb44ef8b0aeea63a43bf49e: Deleting consensus metadata
I20260923 20:25:45.323271 30893 catalog_manager.cc:4841] TS 4bd98534c84c49e985134570c14c8b21 (127.14.127.195:35885): tablet c68672ad3cb145e7aad58ef8bf51fd61 (table test-table [id=15602de8cda1488cbf1e9ee15bb9ae00]) successfully deleted
I20260923 20:25:45.324069 30891 catalog_manager.cc:4841] TS 6949356cbfb44ef8b0aeea63a43bf49e (127.14.127.194:34589): tablet c68672ad3cb145e7aad58ef8bf51fd61 (table test-table [id=15602de8cda1488cbf1e9ee15bb9ae00]) successfully deleted
I20260923 20:25:45.440897 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 30943
W20260923 20:25:45.459486 30891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.127.193:33351 (error 108)
I20260923 20:25:45.459779 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 31095
I20260923 20:25:45.461898 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.461322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:54178)
0923 20:25:45.461599 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.461717 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:33351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260923 20:25:45.462182 30891 catalog_manager.cc:4584] TS 9055e0d5f7fd4c2ebf792773d9b95e15 (127.14.127.193:33351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c68672ad3cb145e7aad58ef8bf51fd61: Network error: Client connection negotiation failed: client connection to 127.14.127.193:33351: connect: Connection refused (error 111)
I20260923 20:25:45.479354 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.460606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.14.127.194:34589)
0923 20:25:45.460828 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.460835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.460970 (+ 135us) server_negotiation.cc:407] Connection header received
0923 20:25:45.461195 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.461201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.461281 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.461411 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:45.463049 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.463880 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.465241 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.465706 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.465860 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.472658 (+ 6798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:45.472675 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:45.472692 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:45.472735 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:45.476225 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.477089 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.477097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.477099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.477289 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.477678 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.477684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.477687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.478058 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:45.478412 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.478742 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.478968 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 20:25:45.480018 31218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:45.515942 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.515459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:54180)
0923 20:25:45.515691 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.515787 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:33351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 20:25:45.517180 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.516755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:54188)
0923 20:25:45.516929 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.517037 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:33351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 20:25:45.577922 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.577467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:54194)
0923 20:25:45.577677 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.577770 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:33351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:25:45.579124 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.578679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:54204)
0923 20:25:45.578874 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.578969 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:33351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 20:25:45.598920 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 31095
I20260923 20:25:45.618533 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 31226
I20260923 20:25:45.631711 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.619409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.14.127.195:35885)
0923 20:25:45.619607 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:25:45.619613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:25:45.619635 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:25:45.619765 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:25:45.619771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:25:45.619846 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:25:45.619948 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:25:45.621224 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.621945 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.623274 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:25:45.623673 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:25:45.623769 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:25:45.626187 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:25:45.626200 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:25:45.626217 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:25:45.626254 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:25:45.629153 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.629796 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.629802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.629805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.629959 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:25:45.630265 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:25:45.630268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:25:45.630270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:25:45.630606 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:25:45.630889 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0923 20:25:45.631179 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 20:25:45.631346 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:25:45.632392 31349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:45.660356 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.659868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:54208)
0923 20:25:45.660098 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.660194 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:33351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 20:25:45.661489 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:25:45.661103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:33351 (local address 127.0.0.1:54214)
0923 20:25:45.661305 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:25:45.661383 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:33351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 20:25:45.722254 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 31226
I20260923 20:25:45.739877 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 30873
I20260923 20:25:45.741041 30936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:25:45.856762 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 30873
I20260923 20:26:05.879132 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:32835
--webserver_interface=127.14.127.254
--webserver_port=35547
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:06.235890 31403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:06.236167 31403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:06.236338 31403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:06.243435 31403 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:26:06.243525 31403 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:26:06.243618 31403 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:26:06.243714 31403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:06.243818 31403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:06.243939 31403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:06.244033 31403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:06.244113 31403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:06.253829 31403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=35547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:06.255132 31403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:06.256486 31403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:06.268550 31408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:06.268625 31411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:06.268591 31409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:06.269227 31403 server_base.cc:1034] running on GCE node
I20260923 20:26:06.270093 31403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:06.272122 31403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:06.273382 31403 hybrid_clock.cc:648] HybridClock initialized: now 1790195166273304 us; error 40 us; skew 500 ppm
I20260923 20:26:06.273881 31403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:06 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:06.283244 31403 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:26:06.284713 31403 webserver.cc:460] Webserver started at http://127.14.127.254:35547/ using document root <none> and password file <none>
I20260923 20:26:06.285437 31403 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:06.285563 31403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:06.291023 31403 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:06.293689 31418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:06.295029 31403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:26:06.295212 31403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "8a4a0a2999b44936ba4d9b04f37f9eec"
format_stamp: "Formatted at 2026-09-23 20:25:20 on dist-test-slave-wmd0"
server_key: "ee42adf781353a2627a4668e08683a25"
server_key_iv: "9c40da1e8c197da3fdeb85426833b893"
server_key_version: "encryptionkey@0"
I20260923 20:26:06.296147 31403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:06.315459 31403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:06.319303 31403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:06.319590 31403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:06.337946 31403 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:32835
I20260923 20:26:06.337983 31469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:32835 every 8 connection(s)
I20260923 20:26:06.339550 31403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:06.342496 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 31403
I20260923 20:26:06.343351 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:33351
--local_ip_for_outbound_sockets=127.14.127.193
--tserver_master_addrs=127.14.127.254:32835
--webserver_port=36609
--webserver_interface=127.14.127.193
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:26:06.346253 31470 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:26:06.349388 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Bootstrap starting.
I20260923 20:26:06.379827 31470 log.cc:824] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:06.395852 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:26:06.396395 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Bootstrap complete.
I20260923 20:26:06.405745 31470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:26:06.406667 31470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4a0a2999b44936ba4d9b04f37f9eec, State: Initialized, Role: FOLLOWER
I20260923 20:26:06.407337 31470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:26:06.407598 31470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:06.407720 31470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:06.407912 31470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:26:06.411651 31470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:26:06.412187 31470 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a4a0a2999b44936ba4d9b04f37f9eec; no voters:
I20260923 20:26:06.412879 31470 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 20:26:06.413017 31474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:26:06.414119 31474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [term 3 LEADER]: Becoming Leader. State: Replica: 8a4a0a2999b44936ba4d9b04f37f9eec, State: Running, Role: LEADER
I20260923 20:26:06.414747 31474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } }
I20260923 20:26:06.415071 31470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:06.417188 31476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4a0a2999b44936ba4d9b04f37f9eec. Latest consensus state: current_term: 3 leader_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } } }
I20260923 20:26:06.417243 31475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a0a2999b44936ba4d9b04f37f9eec" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 32835 } } }
I20260923 20:26:06.417857 31476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:06.417922 31475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:06.431877 31480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:06.440271 31480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:06.442076 31480 catalog_manager.cc:1259] Loaded cluster ID: eaef0f26cbd5438790cd497dff794cff
I20260923 20:26:06.442229 31480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:06.446580 31480 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:06.448279 31480 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Loaded TSK: 0
I20260923 20:26:06.448522 31480 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a4a0a2999b44936ba4d9b04f37f9eec: Loaded TSK: 1
F20260923 20:26:06.449717 31480 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f33804ae9ad google::LogMessage::Fail() at ??:0
@ 0x7f33804b188a google::LogMessage::SendToLog() at ??:0
@ 0x7f33804ae4de google::LogMessage::Flush() at ??:0
@ 0x7f33804afa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f33842127d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f338420173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3395edd1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3395edc783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3395f682a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f3395f680c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f3395fb25e9 std::function<>::operator()() at ??:0
@ 0x7f3395ee0cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3395edec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3395f66610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3395f66439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f33830fa345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f33830fd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f33830fd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f33830d8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f33863b26db start_thread at ??:0
@ 0x7f337f34861f clone at ??:0
W20260923 20:26:06.765450 31472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:06.765759 31472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:06.765857 31472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:06.773464 31472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:06.773586 31472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:06.773702 31472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:26:06.784032 31472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:33351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=36609
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:06.785295 31472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:06.786577 31472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:06.800308 31517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:06.800343 31520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:06.800398 31518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:06.801427 31472 server_base.cc:1034] running on GCE node
I20260923 20:26:06.802090 31472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:06.803296 31472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:06.804520 31472 hybrid_clock.cc:648] HybridClock initialized: now 1790195166804442 us; error 79 us; skew 500 ppm
I20260923 20:26:06.804850 31472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:06 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195166, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:06.808307 31472 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:26:06.809808 31472 webserver.cc:460] Webserver started at http://127.14.127.193:36609/ using document root <none> and password file <none>
I20260923 20:26:06.810495 31472 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:06.810655 31472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:06.816040 31472 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 20:26:06.819029 31527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:06.820410 31472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:26:06.820580 31472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "9055e0d5f7fd4c2ebf792773d9b95e15"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "ad4bb5c5c5309bbdfde13e821dde93b7"
server_key_iv: "11f847fd3465290f8d937728ad96e70d"
server_key_version: "encryptionkey@0"
I20260923 20:26:06.821527 31472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:06.836197 31472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:06.840728 31472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:06.840998 31472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:06.841964 31472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:06.843788 31534 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260923 20:26:06.849063 31534 ts_tablet_manager.cc:1852] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 20:26:06.850096 31534 ts_tablet_manager.cc:1905] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:06.857254 31534 ts_tablet_manager.cc:1918] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:06.857496 31534 log.cc:1196] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/c68672ad3cb145e7aad58ef8bf51fd61
I20260923 20:26:06.857964 31534 ts_tablet_manager.cc:1939] T c68672ad3cb145e7aad58ef8bf51fd61 P 9055e0d5f7fd4c2ebf792773d9b95e15: Deleting consensus metadata
I20260923 20:26:06.858364 31472 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 20:26:06.858492 31472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260923 20:26:06.858666 31472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:06.858760 31472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:06.892697 31472 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:33351
I20260923 20:26:06.892752 31640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:33351 every 8 connection(s)
I20260923 20:26:06.894069 31472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:26:06.898993 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:06.897109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:57421)
0923 20:26:06.897825 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:06.898064 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":276,"threads_started":1}
W20260923 20:26:06.899577 31641 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:32835 (0 consecutive failures): Network error: Failed to ping master at 127.14.127.254:32835: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
I20260923 20:26:06.900398 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:06.900081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:38755)
0923 20:26:06.900227 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:06.900303 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 20:26:06.901253 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:06.900884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:35809)
0923 20:26:06.901019 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:06.901097 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:26:06.901542 31641 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:26:06.901795 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 31472
I20260923 20:26:06.902406 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:34589
--local_ip_for_outbound_sockets=127.14.127.194
--tserver_master_addrs=127.14.127.254:32835
--webserver_port=43273
--webserver_interface=127.14.127.194
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:07.228982 31644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:07.229266 31644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:07.229421 31644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:07.236549 31644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:07.236657 31644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:07.236846 31644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:26:07.246340 31644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:34589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=43273
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:07.247663 31644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:07.248962 31644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:07.262209 31652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:07.262084 31649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:07.262272 31650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:07.263370 31644 server_base.cc:1034] running on GCE node
I20260923 20:26:07.264055 31644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:07.265021 31644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:07.266250 31644 hybrid_clock.cc:648] HybridClock initialized: now 1790195167266201 us; error 41 us; skew 500 ppm
I20260923 20:26:07.266569 31644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:07 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:07.269685 31644 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:26:07.271209 31644 webserver.cc:460] Webserver started at http://127.14.127.194:43273/ using document root <none> and password file <none>
I20260923 20:26:07.271896 31644 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:07.272043 31644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:07.277586 31644 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:26:07.280581 31659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:07.282027 31644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:26:07.282240 31644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "6949356cbfb44ef8b0aeea63a43bf49e"
format_stamp: "Formatted at 2026-09-23 20:25:21 on dist-test-slave-wmd0"
server_key: "69f8f3b786b250ac60c7dff87f136b7a"
server_key_iv: "25b14fb9333d59abc423d5cd0e84b6ab"
server_key_version: "encryptionkey@0"
I20260923 20:26:07.283409 31644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:07.294786 31644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:07.298514 31644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:07.298815 31644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:07.299767 31644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:07.301119 31644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:07.301254 31644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:07.301386 31644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:07.301472 31644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:07.335320 31644 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:34589
I20260923 20:26:07.335439 31771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:34589 every 8 connection(s)
I20260923 20:26:07.336840 31644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:26:07.341739 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:07.339905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:38381)
0923 20:26:07.340649 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:07.340885 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":303,"threads_started":1}
W20260923 20:26:07.342397 31772 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:32835 (0 consecutive failures): Network error: Failed to ping master at 127.14.127.254:32835: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
I20260923 20:26:07.343267 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:07.342955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:56601)
0923 20:26:07.343088 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:07.343155 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 20:26:07.344023 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:07.343740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:42779)
0923 20:26:07.343852 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:07.343920 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 20:26:07.344275 31772 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:26:07.345876 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 31644
I20260923 20:26:07.346503 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:35885
--local_ip_for_outbound_sockets=127.14.127.195
--tserver_master_addrs=127.14.127.254:32835
--webserver_port=40377
--webserver_interface=127.14.127.195
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:07.677621 31775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:07.677894 31775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:07.677986 31775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:07.685030 31775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:07.685141 31775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:07.685259 31775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:26:07.695142 31775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:35885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=40377
--tserver_master_addrs=127.14.127.254:32835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:07.696348 31775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:07.697592 31775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:07.711081 31780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:07.711042 31783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:07.711143 31781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:07.712111 31775 server_base.cc:1034] running on GCE node
I20260923 20:26:07.712702 31775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:07.713598 31775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:07.714818 31775 hybrid_clock.cc:648] HybridClock initialized: now 1790195167714780 us; error 27 us; skew 500 ppm
I20260923 20:26:07.715127 31775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:07 dist-test-slave-wmd0 krb5kdc[30323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195167, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:07.718114 31775 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:26:07.719594 31775 webserver.cc:460] Webserver started at http://127.14.127.195:40377/ using document root <none> and password file <none>
I20260923 20:26:07.720286 31775 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:07.720432 31775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:07.725665 31775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:26:07.728471 31790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:07.729811 31775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:26:07.729997 31775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "4bd98534c84c49e985134570c14c8b21"
format_stamp: "Formatted at 2026-09-23 20:25:22 on dist-test-slave-wmd0"
server_key: "24fdf737ac3b233fcf00aa46d3c400a9"
server_key_iv: "1acce4ce8024d3f19fd0f33962639a86"
server_key_version: "encryptionkey@0"
I20260923 20:26:07.731033 31775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:07.748536 31775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:07.752540 31775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:07.752827 31775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:07.753798 31775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:07.755175 31775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:07.755307 31775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:07.755445 31775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:07.755540 31775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:07.789176 31775 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:35885
I20260923 20:26:07.789227 31902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:35885 every 8 connection(s)
I20260923 20:26:07.790686 31775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:26:07.795439 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:07.793600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:42425)
0923 20:26:07.794332 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:07.794559 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":287,"threads_started":1}
W20260923 20:26:07.796142 31903 heartbeater.cc:644] Failed to heartbeat to 127.14.127.254:32835 (0 consecutive failures): Network error: Failed to ping master at 127.14.127.254:32835: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
I20260923 20:26:07.796969 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:07.796615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:57767)
0923 20:26:07.796784 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:07.796865 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:26:07.797786 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:07.797461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:38553)
0923 20:26:07.797600 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:07.797680 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 20:26:07.798054 31903 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:26:07.799405 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 31775
I20260923 20:26:07.902894 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:07.902069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:47425)
0923 20:26:07.902485 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:07.902583 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":223,"threads_started":1}
I20260923 20:26:08.345739 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:08.344837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:50165)
0923 20:26:08.345391 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:08.345480 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":312,"threads_started":1}
I20260923 20:26:08.799477 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:08.798680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:60717)
0923 20:26:08.799135 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:08.799256 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":214,"threads_started":1}
I20260923 20:26:08.904593 31910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:08.903802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:34255)
0923 20:26:08.904314 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:08.904417 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":304,"threads_started":1}
I20260923 20:26:09.347350 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:09.346567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:41355)
0923 20:26:09.347101 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:09.347193 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":267,"threads_started":1}
I20260923 20:26:09.800981 31912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:09.800314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:37857)
0923 20:26:09.800747 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:09.800823 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260923 20:26:09.906116 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:09.905407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:35929)
0923 20:26:09.905853 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:09.905937 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":224,"threads_started":1}
I20260923 20:26:10.348920 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:10.348226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:33927)
0923 20:26:10.348664 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:10.348748 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
I20260923 20:26:10.802666 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:10.801879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:58473)
0923 20:26:10.802348 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:10.802432 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":234,"threads_started":1}
I20260923 20:26:10.907673 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:10.906978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:43625)
0923 20:26:10.907421 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:10.907514 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":222,"threads_started":1}
I20260923 20:26:11.350488 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:11.349764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:37503)
0923 20:26:11.350227 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:11.350310 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
I20260923 20:26:11.804255 31918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:11.803506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:34071)
0923 20:26:11.803970 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:11.804050 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":253,"threads_started":1}
I20260923 20:26:11.909355 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:11.908540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:45283)
0923 20:26:11.909088 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:11.909174 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":317,"threads_started":1}
I20260923 20:26:12.352137 31920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:12.351406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:46497)
0923 20:26:12.351884 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:12.351966 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":268,"threads_started":1}
I20260923 20:26:12.805939 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:12.805123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:50867)
0923 20:26:12.805689 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:12.805770 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":335,"threads_started":1}
I20260923 20:26:12.911031 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:12.910296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:42377)
0923 20:26:12.910789 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:12.910873 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":235,"threads_started":1}
I20260923 20:26:13.353883 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:13.353157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:55323)
0923 20:26:13.353633 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:13.353726 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
I20260923 20:26:13.807623 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:13.806905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:53345)
0923 20:26:13.807374 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:13.807464 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":262,"threads_started":1}
I20260923 20:26:13.912830 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:13.911982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:44261)
0923 20:26:13.912506 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:13.912605 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":252,"threads_started":1}
I20260923 20:26:14.355789 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:14.354927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:33717)
0923 20:26:14.355465 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:14.355564 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":235,"threads_started":1}
I20260923 20:26:14.809608 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:14.808735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:53629)
0923 20:26:14.809298 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:14.809391 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":254,"threads_started":1}
I20260923 20:26:14.914590 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:14.913865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:34993)
0923 20:26:14.914331 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:14.914422 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":251,"threads_started":1}
I20260923 20:26:15.357722 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:15.356936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:60741)
0923 20:26:15.357434 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:15.357535 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":262,"threads_started":1}
I20260923 20:26:15.811494 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:15.810741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:36093)
0923 20:26:15.811245 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:15.811332 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
I20260923 20:26:15.916257 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:15.915501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:49067)
0923 20:26:15.915988 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:15.916082 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":248,"threads_started":1}
I20260923 20:26:16.359613 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:16.358863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:40611)
0923 20:26:16.359352 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:16.359439 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":239,"threads_started":1}
I20260923 20:26:16.813273 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:16.812440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:59613)
0923 20:26:16.812993 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:16.813086 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":263,"threads_started":1}
I20260923 20:26:16.917968 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:16.917232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:50445)
0923 20:26:16.917699 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:16.917802 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":253,"threads_started":1}
I20260923 20:26:17.361312 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:17.360596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:46945)
0923 20:26:17.361054 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:17.361141 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260923 20:26:17.815449 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:17.814492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:59825)
0923 20:26:17.815131 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:17.815274 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":399,"threads_started":1}
I20260923 20:26:17.919714 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:17.918973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:36247)
0923 20:26:17.919428 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:17.919527 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":252,"threads_started":1}
I20260923 20:26:18.362983 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:18.362237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:40259)
0923 20:26:18.362711 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:18.362797 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260923 20:26:18.817174 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:18.816440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:37699)
0923 20:26:18.816889 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:18.816987 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":219,"threads_started":1}
I20260923 20:26:18.921355 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:18.920621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:36427)
0923 20:26:18.921119 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:18.921198 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":308,"threads_started":1}
I20260923 20:26:19.364698 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:19.363941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:37605)
0923 20:26:19.364448 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:19.364534 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":272,"threads_started":1}
I20260923 20:26:19.819079 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:19.818254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:54891)
0923 20:26:19.818815 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:19.818923 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":261,"threads_started":1}
I20260923 20:26:19.923101 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:19.922265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:51869)
0923 20:26:19.922812 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:19.922917 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":228,"threads_started":1}
I20260923 20:26:20.366674 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:20.365790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:32921)
0923 20:26:20.366310 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:20.366440 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":284,"threads_started":1}
I20260923 20:26:20.820943 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:20.819997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:43073)
0923 20:26:20.820579 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:20.820675 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":371,"threads_started":1}
I20260923 20:26:20.924794 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:20.923964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:48919)
0923 20:26:20.924503 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:20.924589 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":301,"threads_started":1}
I20260923 20:26:21.368203 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:21.367502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:45521)
0923 20:26:21.367919 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:21.367998 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":223,"threads_started":1}
I20260923 20:26:21.822814 31948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:21.821963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:53893)
0923 20:26:21.822458 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:21.822552 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":249,"threads_started":1}
I20260923 20:26:21.926676 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:21.925870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.193:41997)
0923 20:26:21.926344 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:21.926437 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":261,"threads_started":1}
I20260923 20:26:22.370276 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:22.369399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.194:48243)
0923 20:26:22.369966 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:22.370086 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
I20260923 20:26:22.802790 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 31472
I20260923 20:26:22.821131 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:22.803891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.14.127.193:33351)
0923 20:26:22.804678 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:22.804697 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:22.804766 (+ 69us) server_negotiation.cc:407] Connection header received
0923 20:26:22.806601 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:22.806675 (+ 74us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:22.807134 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:22.807485 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:22.808986 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:22.810200 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:22.811539 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:22.812195 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:22.812392 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:22.815089 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:22.815097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:22.815112 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:22.815149 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:22.817834 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:22.818511 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:22.818519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:22.818534 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:22.818768 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:22.819147 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:22.819152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:22.819157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:22.819676 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:22.819997 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:22.820413 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 20:26:22.820847 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":412,"threads_started":1}
I20260923 20:26:22.822503 31636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:22.827491 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:22.823958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:32835 (local address 127.14.127.195:41297)
0923 20:26:22.827135 (+ 3177us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:22.827277 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.254:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2997,"thread_start_us":2864,"threads_started":1}
I20260923 20:26:22.909804 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 31472
I20260923 20:26:22.927465 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 31644
I20260923 20:26:22.945091 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:22.928415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.14.127.194:34589)
0923 20:26:22.928938 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:22.928951 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:22.928987 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:26:22.930571 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:22.930581 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:22.931094 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:22.931451 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:22.932938 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:22.934301 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:22.935757 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:22.936432 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:22.936671 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:22.939276 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:22.939283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:22.939298 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:22.939340 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:22.942096 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:22.942762 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:22.942768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:22.942774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:22.942937 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:22.943256 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:22.943259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:22.943261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:22.943776 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:22.944010 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:22.944366 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 20:26:22.944803 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260923 20:26:22.946466 31767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:23.030746 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 31644
I20260923 20:26:23.048271 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 31775
I20260923 20:26:23.066661 31956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:23.049328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.14.127.195:35885)
0923 20:26:23.049917 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:23.049931 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:23.049975 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:26:23.051709 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:23.051720 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:23.052147 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:23.052505 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:23.053958 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:23.055329 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:23.056645 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:23.057296 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:23.057497 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:23.060258 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:23.060266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:23.060282 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:23.060319 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:23.063309 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:23.064004 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:23.064018 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:23.064025 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:23.064197 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:23.064578 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:23.064584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:23.064587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:23.065113 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:23.065419 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:23.065897 (+ 478us) server_negotiation.cc:299] Negotiation successful
0923 20:26:23.066309 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":297,"threads_started":1}
I20260923 20:26:23.068092 31898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:23.151738 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 31775
2026-09-23T20:26:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64315 ms)
[----------] 30 tests from SecurityITest (300362 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:23 dist-test-slave-wmd0 krb5kdc[31960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:23 dist-test-slave-wmd0 krb5kdc[31960](info): set up 2 sockets
Sep 23 20:26:23 dist-test-slave-wmd0 krb5kdc[31960](info): commencing operation
krb5kdc: starting...
W20260923 20:26:24.266894 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:24 dist-test-slave-wmd0 krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:26:24.511550 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:38319
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:38319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:24.846822 31973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:24.847177 31973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:24.847333 31973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:24.854630 31973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:24.854770 31973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:24.854859 31973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:24.854964 31973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:24.855047 31973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:24.864871 31973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:38319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:38319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:24.866248 31973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:24.867630 31973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:24.879948 31979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:24.880049 31978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:24.880071 31981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:24.880359 31973 server_base.cc:1034] running on GCE node
I20260923 20:26:24.881321 31973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:24.882685 31973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:24.883888 31973 hybrid_clock.cc:648] HybridClock initialized: now 1790195184883827 us; error 52 us; skew 500 ppm
I20260923 20:26:24.884246 31973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:24 dist-test-slave-wmd0 krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195184, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:24.892504 31973 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:26:24.893934 31973 webserver.cc:460] Webserver started at http://127.14.127.254:35161/ using document root <none> and password file <none>
I20260923 20:26:24.894578 31973 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:24.894757 31973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:24.895017 31973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:24.897883 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "162f12f12b7d452fa3f3131b4abac7c4"
format_stamp: "Formatted at 2026-09-23 20:26:24 on dist-test-slave-wmd0"
server_key: "1ba1510e5068dc13deecb599e62de5a6"
server_key_iv: "7de8e1c282dedf230415f441fb2b564f"
server_key_version: "encryptionkey@0"
I20260923 20:26:24.898566 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "162f12f12b7d452fa3f3131b4abac7c4"
format_stamp: "Formatted at 2026-09-23 20:26:24 on dist-test-slave-wmd0"
server_key: "1ba1510e5068dc13deecb599e62de5a6"
server_key_iv: "7de8e1c282dedf230415f441fb2b564f"
server_key_version: "encryptionkey@0"
I20260923 20:26:24.903776 31973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:26:24.907094 31988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:24.908401 31973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:24.908586 31973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "162f12f12b7d452fa3f3131b4abac7c4"
format_stamp: "Formatted at 2026-09-23 20:26:24 on dist-test-slave-wmd0"
server_key: "1ba1510e5068dc13deecb599e62de5a6"
server_key_iv: "7de8e1c282dedf230415f441fb2b564f"
server_key_version: "encryptionkey@0"
I20260923 20:26:24.908795 31973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:24.937186 31973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:24.941267 31973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:24.941562 31973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:24.960181 31973 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:38319
I20260923 20:26:24.960206 32039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:38319 every 8 connection(s)
I20260923 20:26:24.961576 31973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:24.965401 32040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:24.966044 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 31973
I20260923 20:26:24.966259 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:24.966972 14847 external_mini_cluster.cc:1428] Setting key 318b7b247a42f639f4c69fb3cc07cf8c
I20260923 20:26:24.979590 32040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4: Bootstrap starting.
Sep 23 20:26:24 dist-test-slave-wmd0 krb5kdc[31960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:24.983340 32040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:24.984424 32040 log.cc:824] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:24.987445 32040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4: No bootstrap required, opened a new log
I20260923 20:26:24.990531 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:24.968958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.14.127.254:38319)
0923 20:26:24.969768 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:24.969779 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:24.969820 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:26:24.971585 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:24.971611 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:24.971999 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:24.972341 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:24.974222 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:24.975562 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:24.977468 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:24.978110 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:24.978267 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:24.983255 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:24.983263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:24.983277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:24.983312 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:24.986539 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:24.987276 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:24.987282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:24.987288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:24.987447 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:24.987847 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:24.987853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:24.987866 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:24.988363 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:24.988664 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:24.989140 (+ 476us) server_negotiation.cc:299] Negotiation successful
0923 20:26:24.989545 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":382,"threads_started":1}
I20260923 20:26:24.995477 32040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162f12f12b7d452fa3f3131b4abac7c4" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38319 } }
I20260923 20:26:24.995849 32040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:24.995973 32040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162f12f12b7d452fa3f3131b4abac7c4, State: Initialized, Role: FOLLOWER
I20260923 20:26:24.996527 32040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162f12f12b7d452fa3f3131b4abac7c4" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38319 } }
I20260923 20:26:24.996783 32040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:24.996919 32040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:24.997107 32040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:24.999717 32040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162f12f12b7d452fa3f3131b4abac7c4" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38319 } }
I20260923 20:26:25.000231 32040 leader_election.cc:302] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 162f12f12b7d452fa3f3131b4abac7c4; no voters:
I20260923 20:26:25.000813 32040 leader_election.cc:288] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:25.000945 32045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:25.002322 32045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [term 1 LEADER]: Becoming Leader. State: Replica: 162f12f12b7d452fa3f3131b4abac7c4, State: Running, Role: LEADER
I20260923 20:26:25.002933 32045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162f12f12b7d452fa3f3131b4abac7c4" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38319 } }
I20260923 20:26:25.003113 32040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:25.006012 32047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 162f12f12b7d452fa3f3131b4abac7c4. Latest consensus state: current_term: 1 leader_uuid: "162f12f12b7d452fa3f3131b4abac7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162f12f12b7d452fa3f3131b4abac7c4" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38319 } } }
I20260923 20:26:25.006150 32046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "162f12f12b7d452fa3f3131b4abac7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162f12f12b7d452fa3f3131b4abac7c4" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38319 } } }
I20260923 20:26:25.006650 32046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:25.006675 32047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:25.008083 32054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:25.012840 32054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:25.021479 32054 catalog_manager.cc:1347] Generated new cluster ID: 858d18b4651945558930adc2d2194221
I20260923 20:26:25.021598 32054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:25.032235 32054 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:25.034106 32054 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:25.041730 32054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4: Generated new TSK 0
I20260923 20:26:25.042527 32054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:26:25.100196 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:38319
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:25.432972 32066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:25.433257 32066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:25.433413 32066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:25.440553 32066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:25.440652 32066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:25.440878 32066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:26:25.451548 32066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:38319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:25.452958 32066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:25.454361 32066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:25.467631 32071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:25.467715 32072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:25.467685 32074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:25.468852 32066 server_base.cc:1034] running on GCE node
I20260923 20:26:25.469497 32066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:25.470475 32066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:25.471719 32066 hybrid_clock.cc:648] HybridClock initialized: now 1790195185471662 us; error 73 us; skew 500 ppm
I20260923 20:26:25.472079 32066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:25 dist-test-slave-wmd0 krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195185, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:25.475601 32066 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:26:25.477114 32066 webserver.cc:460] Webserver started at http://127.14.127.193:45945/ using document root <none> and password file <none>
I20260923 20:26:25.477859 32066 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:25.477994 32066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:25.478331 32066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:25.481374 32066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "4126be778c7f48a68fe99c87c09ff7a1"
format_stamp: "Formatted at 2026-09-23 20:26:25 on dist-test-slave-wmd0"
server_key: "04df52bb7b2d41b3fe7935c80f4480ff"
server_key_iv: "27343262e53b496bac01107f01b4d09a"
server_key_version: "encryptionkey@0"
I20260923 20:26:25.482167 32066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "4126be778c7f48a68fe99c87c09ff7a1"
format_stamp: "Formatted at 2026-09-23 20:26:25 on dist-test-slave-wmd0"
server_key: "04df52bb7b2d41b3fe7935c80f4480ff"
server_key_iv: "27343262e53b496bac01107f01b4d09a"
server_key_version: "encryptionkey@0"
I20260923 20:26:25.487493 32066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:26:25.490965 32081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:25.492306 32066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:25.492480 32066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "4126be778c7f48a68fe99c87c09ff7a1"
format_stamp: "Formatted at 2026-09-23 20:26:25 on dist-test-slave-wmd0"
server_key: "04df52bb7b2d41b3fe7935c80f4480ff"
server_key_iv: "27343262e53b496bac01107f01b4d09a"
server_key_version: "encryptionkey@0"
I20260923 20:26:25.492699 32066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:25.514494 32066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:25.518786 32066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:25.519086 32066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:25.520077 32066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:25.521433 32066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:25.521512 32066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:25.521695 32066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:25.521749 32066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:25.556298 32066 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:32963
I20260923 20:26:25.556339 32193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:32963 every 8 connection(s)
I20260923 20:26:25.557669 32066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:26:25.564894 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 32066
I20260923 20:26:25.565141 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:26:25.565798 14847 external_mini_cluster.cc:1428] Setting key 2ef5789151076b99d4531fe2256eaad5
Sep 23 20:26:25 dist-test-slave-wmd0 krb5kdc[31960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195185, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:25.577936 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:25.560272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:41015 (local address 127.14.127.254:38319)
0923 20:26:25.560729 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:25.560734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:25.561747 (+ 1013us) server_negotiation.cc:407] Connection header received
0923 20:26:25.563703 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:25.563708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:25.563812 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:25.563936 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:25.565567 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:25.566370 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:25.568005 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:25.568401 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:25.568538 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:25.572192 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:25.572198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:25.572201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:25.572234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:25.574846 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:25.575593 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:25.575603 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:25.575605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:25.575731 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:25.576176 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:25.576185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:25.576188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:25.576449 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:25.576606 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:25.577347 (+ 741us) server_negotiation.cc:299] Negotiation successful
0923 20:26:25.577573 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260923 20:26:25.579001 32197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:25.560874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:38319 (local address 127.14.127.193:41015)
0923 20:26:25.561586 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:25.561629 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:25.563423 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:25.564191 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:25.564200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:25.564727 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:25.565394 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:25.565413 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:25.566530 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:25.566534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:25.567853 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:25.567866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:25.568656 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:25.568664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:25.568893 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:25.569596 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:25.569625 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:25.571958 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:25.574994 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:25.575003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:25.575020 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:25.575464 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:25.575844 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:25.575850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:25.575853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:25.576048 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:25.576593 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:25.576601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:25.576951 (+ 350us) client_negotiation.cc:769] Sending connection context
0923 20:26:25.577225 (+ 274us) client_negotiation.cc:240] Negotiation successful
0923 20:26:25.577760 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":282,"threads_started":1}
I20260923 20:26:25.580701 32194 heartbeater.cc:347] Connected to a master server at 127.14.127.254:38319
I20260923 20:26:25.581008 32194 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:25.581749 32194 heartbeater.cc:511] Master 127.14.127.254:38319 requested a full tablet report, sending...
I20260923 20:26:25.584157 32005 ts_manager.cc:194] Registered new tserver with Master: 4126be778c7f48a68fe99c87c09ff7a1 (127.14.127.193:32963)
I20260923 20:26:25.586416 32005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:41015
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:26:25.634111 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:38319
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:25.964804 32200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:25.965085 32200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:25.965250 32200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:25.972565 32200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:25.972667 32200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:25.972848 32200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:26:25.982596 32200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:38319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:25.983937 32200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:25.985265 32200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:25.999109 32206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:25.999193 32205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:25.999136 32208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:25.999918 32200 server_base.cc:1034] running on GCE node
I20260923 20:26:26.000512 32200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:26.001439 32200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:26.002713 32200 hybrid_clock.cc:648] HybridClock initialized: now 1790195186002654 us; error 49 us; skew 500 ppm
I20260923 20:26:26.003110 32200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:26 dist-test-slave-wmd0 krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195186, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:26.006280 32200 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:26:26.007942 32200 webserver.cc:460] Webserver started at http://127.14.127.194:36623/ using document root <none> and password file <none>
I20260923 20:26:26.008638 32200 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:26.008787 32200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:26.009107 32200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:26.012056 32200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "5eb668f2a4e8421b9fb9b8a268397312"
format_stamp: "Formatted at 2026-09-23 20:26:26 on dist-test-slave-wmd0"
server_key: "8a86fe9c7d77233d5f85590d25c7f746"
server_key_iv: "3790308f71b1133b618218fa90430fb0"
server_key_version: "encryptionkey@0"
I20260923 20:26:26.012890 32200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "5eb668f2a4e8421b9fb9b8a268397312"
format_stamp: "Formatted at 2026-09-23 20:26:26 on dist-test-slave-wmd0"
server_key: "8a86fe9c7d77233d5f85590d25c7f746"
server_key_iv: "3790308f71b1133b618218fa90430fb0"
server_key_version: "encryptionkey@0"
I20260923 20:26:26.020224 32200 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260923 20:26:26.023808 32215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:26.025137 32200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:26:26.025323 32200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "5eb668f2a4e8421b9fb9b8a268397312"
format_stamp: "Formatted at 2026-09-23 20:26:26 on dist-test-slave-wmd0"
server_key: "8a86fe9c7d77233d5f85590d25c7f746"
server_key_iv: "3790308f71b1133b618218fa90430fb0"
server_key_version: "encryptionkey@0"
I20260923 20:26:26.025532 32200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:26.060722 32200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:26.064985 32200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:26.065294 32200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:26.066284 32200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:26.067727 32200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:26.067863 32200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:26.068012 32200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:26.068087 32200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:26.102121 32200 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:34057
I20260923 20:26:26.102211 32327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:34057 every 8 connection(s)
I20260923 20:26:26.103655 32200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:26:26.109234 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 32200
I20260923 20:26:26.109444 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:26:26.109920 14847 external_mini_cluster.cc:1428] Setting key a0acd4b6575d091775af73270feddd6c
Sep 23 20:26:26 dist-test-slave-wmd0 krb5kdc[31960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195186, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:26.126492 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.106240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:42839 (local address 127.14.127.254:38319)
0923 20:26:26.106742 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.106747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.107879 (+ 1132us) server_negotiation.cc:407] Connection header received
0923 20:26:26.109865 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.109870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.109962 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.110096 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:26.112028 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.112824 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.115355 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.115791 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.115887 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.120028 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:26.120033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:26.120037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:26.120069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:26.123233 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:26.124179 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:26.124182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:26.124185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:26.124311 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:26.124813 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:26.124816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:26.124818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:26.125078 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:26.125187 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.126070 (+ 883us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.126230 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":224,"threads_started":1}
I20260923 20:26:26.128103 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.106914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:38319 (local address 127.14.127.194:42839)
0923 20:26:26.107713 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:26.107757 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:26.109597 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:26.110384 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:26.110397 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:26.111181 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:26.111842 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.111864 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.112974 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.112982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.115213 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.115232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.116026 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.116034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.116356 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.117083 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:26.117123 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:26.119783 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:26.123376 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:26.123388 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:26.123408 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:26.124051 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:26.124428 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:26.124436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:26.124442 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:26.124701 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:26.125212 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:26.125223 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:26.125696 (+ 473us) client_negotiation.cc:769] Sending connection context
0923 20:26:26.125995 (+ 299us) client_negotiation.cc:240] Negotiation successful
0923 20:26:26.126565 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":322,"threads_started":1}
I20260923 20:26:26.129753 32328 heartbeater.cc:347] Connected to a master server at 127.14.127.254:38319
I20260923 20:26:26.130113 32328 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:26.130993 32328 heartbeater.cc:511] Master 127.14.127.254:38319 requested a full tablet report, sending...
I20260923 20:26:26.132501 32005 ts_manager.cc:194] Registered new tserver with Master: 5eb668f2a4e8421b9fb9b8a268397312 (127.14.127.194:34057)
I20260923 20:26:26.133821 32005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:42839
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:26:26.166563 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:38319
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:26.499176 32334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:26.499460 32334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:26.499562 32334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:26.507289 32334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:26.507404 32334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:26.507522 32334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:26:26.517959 32334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:38319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:26.519297 32334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:26.520536 32334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:26.533679 32342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:26.533782 32339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:26.533764 32340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:26.534911 32334 server_base.cc:1034] running on GCE node
I20260923 20:26:26.535548 32334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:26.536468 32334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:26.537730 32334 hybrid_clock.cc:648] HybridClock initialized: now 1790195186537684 us; error 53 us; skew 500 ppm
I20260923 20:26:26.538115 32334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:26 dist-test-slave-wmd0 krb5kdc[31960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195186, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:26.541263 32334 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:26:26.542809 32334 webserver.cc:460] Webserver started at http://127.14.127.195:33731/ using document root <none> and password file <none>
I20260923 20:26:26.543488 32334 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:26.543635 32334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:26.543947 32334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:26.547008 32334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "3254d0a14cf64879ae5cb34a3b49ca28"
format_stamp: "Formatted at 2026-09-23 20:26:26 on dist-test-slave-wmd0"
server_key: "39591b5145f3381418e56664496b6c0c"
server_key_iv: "f470e0eeaf160b5d7be04e35d6e03c52"
server_key_version: "encryptionkey@0"
I20260923 20:26:26.547786 32334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "3254d0a14cf64879ae5cb34a3b49ca28"
format_stamp: "Formatted at 2026-09-23 20:26:26 on dist-test-slave-wmd0"
server_key: "39591b5145f3381418e56664496b6c0c"
server_key_iv: "f470e0eeaf160b5d7be04e35d6e03c52"
server_key_version: "encryptionkey@0"
I20260923 20:26:26.553073 32334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 20:26:26.557219 32349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:26.558807 32334 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 20:26:26.558985 32334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "3254d0a14cf64879ae5cb34a3b49ca28"
format_stamp: "Formatted at 2026-09-23 20:26:26 on dist-test-slave-wmd0"
server_key: "39591b5145f3381418e56664496b6c0c"
server_key_iv: "f470e0eeaf160b5d7be04e35d6e03c52"
server_key_version: "encryptionkey@0"
I20260923 20:26:26.559204 32334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:26.578256 32334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:26.582233 32334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:26.582554 32334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:26.583547 32334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:26.584868 32334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:26.584976 32334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:26.585137 32334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:26.585217 32334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:26.589586 32194 heartbeater.cc:503] Master 127.14.127.254:38319 was elected leader, sending a full tablet report...
I20260923 20:26:26.619495 32334 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:41915
I20260923 20:26:26.619514 32461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:41915 every 8 connection(s)
I20260923 20:26:26.620833 32334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:26:26.629763 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 32334
I20260923 20:26:26.629959 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:26:26.630508 14847 external_mini_cluster.cc:1428] Setting key 1373317b6fd9123e32cf4c4e63414626
Sep 23 20:26:26 dist-test-slave-wmd0 krb5kdc[31960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195186, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:26.640671 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.623606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:47075 (local address 127.14.127.254:38319)
0923 20:26:26.623788 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.623792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.624911 (+ 1119us) server_negotiation.cc:407] Connection header received
0923 20:26:26.626784 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.626790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.626889 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.627022 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:26.628582 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.629438 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.631308 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.631743 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.631882 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.635463 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:26.635468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:26.635472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:26.635506 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:26.637897 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:26.638666 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:26.638675 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:26.638683 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:26.638820 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:26.639212 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:26.639216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:26.639218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:26.639497 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:26.639633 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.640250 (+ 617us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.640426 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:26:26.641657 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.624044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:38319 (local address 127.14.127.195:47075)
0923 20:26:26.624741 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:26.624780 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:26.626460 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:26.627271 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:26.627280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:26.627814 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:26.628382 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.628399 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.629617 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.629624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.631127 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.631140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.631994 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.632001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.632239 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.632969 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:26.632998 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:26.635267 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:26.638025 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:26.638033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:26.638046 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:26.638454 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:26.638921 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:26.638924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:26.638927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:26.639104 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:26.639604 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:26.639609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:26.639929 (+ 320us) client_negotiation.cc:769] Sending connection context
0923 20:26:26.640150 (+ 221us) client_negotiation.cc:240] Negotiation successful
0923 20:26:26.640531 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":266,"threads_started":1}
I20260923 20:26:26.643209 32462 heartbeater.cc:347] Connected to a master server at 127.14.127.254:38319
I20260923 20:26:26.643510 32462 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:26.644331 32462 heartbeater.cc:511] Master 127.14.127.254:38319 requested a full tablet report, sending...
I20260923 20:26:26.646090 32004 ts_manager.cc:194] Registered new tserver with Master: 3254d0a14cf64879ae5cb34a3b49ca28 (127.14.127.195:41915)
I20260923 20:26:26.647079 32004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:47075
I20260923 20:26:26.658188 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:26:26.675611 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.662594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.14.127.254:38319)
0923 20:26:26.662815 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.662824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.663195 (+ 371us) server_negotiation.cc:407] Connection header received
0923 20:26:26.663419 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.663424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.663516 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.663661 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:26.665178 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.665915 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.667397 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.667840 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.667980 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.670390 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:26.670403 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:26.670408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:26.670440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:26.672898 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:26.673577 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:26.673585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:26.673588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:26.673718 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:26.674099 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:26.674105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:26.674111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:26.674431 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:26.674751 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.675052 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.675269 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:26:26.679622 32004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:26:26.686513 32004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:26:26.711416 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.700266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.14.127.193:32963)
0923 20:26:26.700795 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.700802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.700824 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:26:26.700898 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.700903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.701136 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.702247 (+ 1111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:26.702533 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.704579 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.708170 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.710213 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.710412 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.710589 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.710929 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.711086 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":959,"threads_started":1}
I20260923 20:26:26.711483 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.699661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:32963 (local address 127.0.0.1:50664)
0923 20:26:26.700333 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:26.700385 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:26.700570 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:26.701399 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:26.701410 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:26.701454 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:26.701955 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.701974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.704792 (+ 2818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.704800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.707914 (+ 3114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.707940 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.710422 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.710431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.710660 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.710719 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 20:26:26.710865 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 20:26:26.711062 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":768,"threads_started":1}
I20260923 20:26:26.715723 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.701833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:34057 (local address 127.0.0.1:36216)
0923 20:26:26.702403 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:26.702419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:26.702526 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:26.707828 (+ 5302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:26.707833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:26.707860 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:26.708322 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.708338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.709870 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.709879 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.712882 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.712896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.714800 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.714807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.715060 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.715096 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:26:26.715205 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:26:26.715329 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":58,"thread_start_us":737,"threads_started":1}
I20260923 20:26:26.715761 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.702395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.14.127.194:34057)
0923 20:26:26.707186 (+ 4791us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.707193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.707215 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:26:26.707345 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.707352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.707613 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.708529 (+ 916us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:26.708741 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.709736 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.713032 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.714450 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.714649 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.714821 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.715266 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.715433 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4632,"thread_start_us":4518,"threads_started":1}
I20260923 20:26:26.715899 32129 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcedd6a13184e08929e56297f36ae14 (DEFAULT_TABLE table=test-table [id=5c79649a06ae491180a32d278120373d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:26.717466 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.701110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:41915 (local address 127.0.0.1:60404)
0923 20:26:26.701604 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:26.701628 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:26.701751 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:26.710990 (+ 9239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:26.710997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:26.711028 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:26.711380 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.711391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.713468 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.713475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.714943 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.714956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.716659 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.716665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.716853 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.716885 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:26:26.716954 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 20:26:26.717095 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":1,"thread_start_us":406,"threads_started":1}
I20260923 20:26:26.717543 32129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcedd6a13184e08929e56297f36ae14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:26.718554 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.701536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.14.127.195:41915)
0923 20:26:26.710361 (+ 8825us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.710372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.710408 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:26:26.710502 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.710511 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.710774 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.711589 (+ 815us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:26.711880 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.713338 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.715104 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.716427 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.717863 (+ 1436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.718012 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.718077 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.718227 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8658,"thread_start_us":8488,"threads_started":1}
I20260923 20:26:26.719316 32263 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcedd6a13184e08929e56297f36ae14 (DEFAULT_TABLE table=test-table [id=5c79649a06ae491180a32d278120373d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:26.720682 32263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcedd6a13184e08929e56297f36ae14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:26.722122 32397 tablet_service.cc:1467] Processing CreateTablet for tablet 6dcedd6a13184e08929e56297f36ae14 (DEFAULT_TABLE table=test-table [id=5c79649a06ae491180a32d278120373d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:26.723548 32397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dcedd6a13184e08929e56297f36ae14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:26.728993 32480 tablet_bootstrap.cc:492] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Bootstrap starting.
I20260923 20:26:26.732106 32480 tablet_bootstrap.cc:654] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:26.733124 32480 log.cc:824] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:26.735589 32482 tablet_bootstrap.cc:492] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Bootstrap starting.
I20260923 20:26:26.736454 32480 tablet_bootstrap.cc:492] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: No bootstrap required, opened a new log
I20260923 20:26:26.736720 32480 ts_tablet_manager.cc:1397] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260923 20:26:26.738543 32483 tablet_bootstrap.cc:492] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Bootstrap starting.
I20260923 20:26:26.739146 32482 tablet_bootstrap.cc:654] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:26.740347 32482 log.cc:824] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:26.741626 32483 tablet_bootstrap.cc:654] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:26.742545 32483 log.cc:824] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:26.743386 32482 tablet_bootstrap.cc:492] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: No bootstrap required, opened a new log
I20260923 20:26:26.743700 32482 ts_tablet_manager.cc:1397] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260923 20:26:26.747395 32480 raft_consensus.cc:348] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.747848 32480 raft_consensus.cc:374] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:26.747994 32480 raft_consensus.cc:729] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4126be778c7f48a68fe99c87c09ff7a1, State: Initialized, Role: FOLLOWER
I20260923 20:26:26.748624 32480 consensus_queue.cc:260] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.749147 32483 tablet_bootstrap.cc:492] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: No bootstrap required, opened a new log
I20260923 20:26:26.749496 32483 ts_tablet_manager.cc:1397] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20260923 20:26:26.750377 32480 ts_tablet_manager.cc:1428] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 20:26:26.753911 32482 raft_consensus.cc:348] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.754383 32482 raft_consensus.cc:374] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:26.754566 32482 raft_consensus.cc:729] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb668f2a4e8421b9fb9b8a268397312, State: Initialized, Role: FOLLOWER
I20260923 20:26:26.755422 32482 consensus_queue.cc:260] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.756764 32328 heartbeater.cc:503] Master 127.14.127.254:38319 was elected leader, sending a full tablet report...
I20260923 20:26:26.757393 32482 ts_tablet_manager.cc:1428] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260923 20:26:26.758008 32483 raft_consensus.cc:348] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.758386 32483 raft_consensus.cc:374] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:26.758494 32483 raft_consensus.cc:729] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3254d0a14cf64879ae5cb34a3b49ca28, State: Initialized, Role: FOLLOWER
I20260923 20:26:26.759088 32483 consensus_queue.cc:260] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.760466 32462 heartbeater.cc:503] Master 127.14.127.254:38319 was elected leader, sending a full tablet report...
I20260923 20:26:26.760906 32483 ts_tablet_manager.cc:1428] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260923 20:26:26.810091 32195 tablet.cc:2369] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:26:26.855770 32329 tablet.cc:2369] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:26:26.860194 32486 raft_consensus.cc:482] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:26:26.860518 32486 raft_consensus.cc:504] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.862345 32486 leader_election.cc:288] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3254d0a14cf64879ae5cb34a3b49ca28 (127.14.127.195:41915), 5eb668f2a4e8421b9fb9b8a268397312 (127.14.127.194:34057)
I20260923 20:26:26.869081 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.862804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:35987 (local address 127.14.127.195:41915)
0923 20:26:26.863066 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.863072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.863377 (+ 305us) server_negotiation.cc:407] Connection header received
0923 20:26:26.863651 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.863657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.863751 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.863842 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:26.864708 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.865443 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.867370 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.868141 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.868327 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.868403 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.868694 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.868820 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:26:26.869267 32489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.862684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:41915 (local address 127.14.127.193:35987)
0923 20:26:26.863265 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:26.863281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:26.863492 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:26.864074 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:26.864082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:26.864130 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:26.864556 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.864566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.865595 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.865600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.867215 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.867228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.868354 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.868360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.868512 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.868536 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:26:26.868658 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 20:26:26.868792 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":20,"spinlock_wait_cycles":36224,"thread_start_us":299,"threads_started":1}
I20260923 20:26:26.869382 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.863594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:36223 (local address 127.14.127.194:34057)
0923 20:26:26.863844 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.863850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.864266 (+ 416us) server_negotiation.cc:407] Connection header received
0923 20:26:26.864475 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.864480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.864553 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.864649 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:26.865153 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.865852 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.867641 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.868440 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.868612 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.868693 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.868988 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.869119 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 20:26:26.869589 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.863397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:34057 (local address 127.14.127.193:36223)
0923 20:26:26.864159 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:26.864177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:26.864342 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:26.864662 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:26.864666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:26.864690 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:26.865002 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.865012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.865994 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.865999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.867483 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:26.867495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.868685 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.868691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:26.868837 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.868858 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:26:26.868967 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:26:26.869088 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":20,"thread_start_us":401,"threads_started":1}
I20260923 20:26:26.870584 32417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcedd6a13184e08929e56297f36ae14" candidate_uuid: "4126be778c7f48a68fe99c87c09ff7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" is_pre_election: true
I20260923 20:26:26.870862 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcedd6a13184e08929e56297f36ae14" candidate_uuid: "4126be778c7f48a68fe99c87c09ff7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb668f2a4e8421b9fb9b8a268397312" is_pre_election: true
I20260923 20:26:26.871146 32417 raft_consensus.cc:2454] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4126be778c7f48a68fe99c87c09ff7a1 in term 0.
I20260923 20:26:26.871413 32283 raft_consensus.cc:2454] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4126be778c7f48a68fe99c87c09ff7a1 in term 0.
I20260923 20:26:26.871848 32084 leader_election.cc:302] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3254d0a14cf64879ae5cb34a3b49ca28, 4126be778c7f48a68fe99c87c09ff7a1; no voters:
I20260923 20:26:26.872222 32486 raft_consensus.cc:2781] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:26:26.872381 32486 raft_consensus.cc:482] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:26:26.872530 32486 raft_consensus.cc:3037] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 0 FOLLOWER]: Advancing to term 1
W20260923 20:26:26.872526 32463 tablet.cc:2369] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:26:26.874765 32486 raft_consensus.cc:504] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.875329 32486 leader_election.cc:288] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [CANDIDATE]: Term 1 election: Requested vote from peers 3254d0a14cf64879ae5cb34a3b49ca28 (127.14.127.195:41915), 5eb668f2a4e8421b9fb9b8a268397312 (127.14.127.194:34057)
I20260923 20:26:26.875823 32417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcedd6a13184e08929e56297f36ae14" candidate_uuid: "4126be778c7f48a68fe99c87c09ff7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3254d0a14cf64879ae5cb34a3b49ca28"
I20260923 20:26:26.875869 32283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dcedd6a13184e08929e56297f36ae14" candidate_uuid: "4126be778c7f48a68fe99c87c09ff7a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eb668f2a4e8421b9fb9b8a268397312"
I20260923 20:26:26.876042 32417 raft_consensus.cc:3037] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:26.876089 32283 raft_consensus.cc:3037] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:26.879281 32417 raft_consensus.cc:2454] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4126be778c7f48a68fe99c87c09ff7a1 in term 1.
I20260923 20:26:26.879287 32283 raft_consensus.cc:2454] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4126be778c7f48a68fe99c87c09ff7a1 in term 1.
I20260923 20:26:26.879683 32084 leader_election.cc:302] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3254d0a14cf64879ae5cb34a3b49ca28, 4126be778c7f48a68fe99c87c09ff7a1; no voters:
I20260923 20:26:26.880059 32486 raft_consensus.cc:2781] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:26.880493 32486 raft_consensus.cc:686] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 1 LEADER]: Becoming Leader. State: Replica: 4126be778c7f48a68fe99c87c09ff7a1, State: Running, Role: LEADER
I20260923 20:26:26.880949 32486 consensus_queue.cc:237] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } }
I20260923 20:26:26.885224 32004 catalog_manager.cc:5495] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4126be778c7f48a68fe99c87c09ff7a1 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "4126be778c7f48a68fe99c87c09ff7a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4126be778c7f48a68fe99c87c09ff7a1" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 32963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 } health_report { overall_health: UNKNOWN } } }
I20260923 20:26:26.906742 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.901690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.14.127.193:32963)
0923 20:26:26.901865 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.901869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.901886 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:26:26.902007 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.902010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.902087 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.902203 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:26:26.902513 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.903331 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.904848 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.905230 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.905320 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.905721 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:26:26.906192 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:26:26.906306 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.906372 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.906430 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:26:26.914661 32417 raft_consensus.cc:1261] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 1 FOLLOWER]: Refusing update from remote peer 4126be778c7f48a68fe99c87c09ff7a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:26.914718 32283 raft_consensus.cc:1261] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 1 FOLLOWER]: Refusing update from remote peer 4126be778c7f48a68fe99c87c09ff7a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:26.915588 32486 consensus_queue.cc:1035] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3254d0a14cf64879ae5cb34a3b49ca28" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 41915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:26.915920 32491 consensus_queue.cc:1035] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eb668f2a4e8421b9fb9b8a268397312" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:26.939662 32004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:26.939934 32004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:26.943454 32004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 162f12f12b7d452fa3f3131b4abac7c4: Sending DeleteTablet for 3 replicas of tablet 6dcedd6a13184e08929e56297f36ae14
I20260923 20:26:26.944515 32129 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcedd6a13184e08929e56297f36ae14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:26 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:50664
I20260923 20:26:26.944644 32397 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcedd6a13184e08929e56297f36ae14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:26 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:60404
I20260923 20:26:26.944775 32263 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dcedd6a13184e08929e56297f36ae14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:26 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:36216
I20260923 20:26:26.947013 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 32066
I20260923 20:26:26.954963 32506 tablet_replica.cc:330] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: stopping tablet replica
I20260923 20:26:26.955476 32506 raft_consensus.cc:2229] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:26.956100 32506 raft_consensus.cc:2258] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:26.956926 32505 tablet_replica.cc:330] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: stopping tablet replica
I20260923 20:26:26.957554 32504 tablet_replica.cc:330] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: stopping tablet replica
I20260923 20:26:26.957940 32504 raft_consensus.cc:2229] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:26:26.958635 32504 raft_consensus.cc:2258] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:26.958977 32506 ts_tablet_manager.cc:1905] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:26.959264 32505 raft_consensus.cc:2229] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:26.959792 32505 raft_consensus.cc:2258] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:26.964327 32505 ts_tablet_manager.cc:1905] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:26.964671 32504 ts_tablet_manager.cc:1905] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:26.970484 32506 ts_tablet_manager.cc:1918] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:26.974131 32504 ts_tablet_manager.cc:1918] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:26.974681 32504 log.cc:1196] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/6dcedd6a13184e08929e56297f36ae14
I20260923 20:26:26.975144 32506 log.cc:1196] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/6dcedd6a13184e08929e56297f36ae14
I20260923 20:26:26.975768 32506 ts_tablet_manager.cc:1939] T 6dcedd6a13184e08929e56297f36ae14 P 5eb668f2a4e8421b9fb9b8a268397312: Deleting consensus metadata
I20260923 20:26:26.977301 32505 ts_tablet_manager.cc:1918] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:26.977465 32505 log.cc:1196] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/6dcedd6a13184e08929e56297f36ae14
I20260923 20:26:26.977883 31991 catalog_manager.cc:4841] TS 5eb668f2a4e8421b9fb9b8a268397312 (127.14.127.194:34057): tablet 6dcedd6a13184e08929e56297f36ae14 (table test-table [id=5c79649a06ae491180a32d278120373d]) successfully deleted
Sep 23 20:26:26 dist-test-slave-wmd0 krb5kdc[31960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:26:26.978665 32505 ts_tablet_manager.cc:1939] T 6dcedd6a13184e08929e56297f36ae14 P 3254d0a14cf64879ae5cb34a3b49ca28: Deleting consensus metadata
I20260923 20:26:26.980593 32504 ts_tablet_manager.cc:1939] T 6dcedd6a13184e08929e56297f36ae14 P 4126be778c7f48a68fe99c87c09ff7a1: Deleting consensus metadata
I20260923 20:26:26.980695 31991 catalog_manager.cc:4841] TS 3254d0a14cf64879ae5cb34a3b49ca28 (127.14.127.195:41915): tablet 6dcedd6a13184e08929e56297f36ae14 (table test-table [id=5c79649a06ae491180a32d278120373d]) successfully deleted
I20260923 20:26:26.982187 31991 catalog_manager.cc:4841] TS 4126be778c7f48a68fe99c87c09ff7a1 (127.14.127.193:32963): tablet 6dcedd6a13184e08929e56297f36ae14 (table test-table [id=5c79649a06ae491180a32d278120373d]) successfully deleted
I20260923 20:26:26.987025 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:26.948793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.14.127.193:32963)
0923 20:26:26.949008 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:26.949017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:26.957188 (+ 8171us) server_negotiation.cc:407] Connection header received
0923 20:26:26.957480 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:26.957485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:26.957569 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:26.957694 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:26.959798 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.961015 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.965242 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:26.965973 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:26.966073 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:26.979664 (+ 13591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:26.979678 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:26.979696 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:26.979747 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:26.983979 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:26.984699 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:26.984708 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:26.984712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:26.984894 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:26.985320 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:26.985325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:26.985329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:26.985804 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:26.986102 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:26.986367 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 20:26:26.986637 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:26:26.987761 32189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:27.086643 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 32066
I20260923 20:26:27.106580 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 32200
Sep 23 20:26:27 dist-test-slave-wmd0 krb5kdc[31960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:26:27.123893 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:27.107381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.14.127.194:34057)
0923 20:26:27.107555 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:27.107569 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:27.107683 (+ 114us) server_negotiation.cc:407] Connection header received
0923 20:26:27.107854 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:27.107860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:27.107949 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:27.108050 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:27.109291 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:27.110421 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:27.111763 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:27.112426 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:27.112532 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:27.117279 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:27.117291 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:27.117308 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:27.117355 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:27.121000 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:27.121717 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:27.121726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:27.121730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:27.121866 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:27.122185 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:27.122190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:27.122193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:27.122723 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:27.123008 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:27.123308 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 20:26:27.123518 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:26:27.124655 32323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:27.213335 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 32200
I20260923 20:26:27.230284 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 32334
Sep 23 20:26:27 dist-test-slave-wmd0 krb5kdc[31960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:26:27.244707 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:27.231090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.14.127.195:41915)
0923 20:26:27.231278 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:27.231283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:27.231301 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:26:27.231415 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:27.231418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:27.231479 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:27.231566 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:27.232851 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:27.233582 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:27.234990 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:27.235377 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:27.235470 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:27.239693 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:27.239708 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:27.239724 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:27.239759 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:27.242308 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:27.242988 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:27.242995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:27.242997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:27.243140 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:27.243449 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:27.243452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:27.243454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:27.243791 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:27.244001 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:27.244245 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:26:27.244407 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:26:27.245370 32457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:27.335294 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 32334
I20260923 20:26:27.352886 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 31973
I20260923 20:26:27.354058 32035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:27.464852 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 31973
2026-09-23T20:26:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4315 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T20:26:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:27Z Disabled control of system clock
I20260923 20:26:27.541036 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:44145
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:26:27.869159 32515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:27.869438 32515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:27.869556 32515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:27.876806 32515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:27.876914 32515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:27.876952 32515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:27.877076 32515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:27.877136 32515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:27.886397 32515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:44145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:27.887789 32515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:27.889559 32515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:27.901422 32523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:27.901472 32520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:27.901439 32521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:27.902678 32515 server_base.cc:1034] running on GCE node
I20260923 20:26:27.903482 32515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:27.904881 32515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:27.906147 32515 hybrid_clock.cc:648] HybridClock initialized: now 1790195187906096 us; error 55 us; skew 500 ppm
I20260923 20:26:27.906507 32515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:27.908820 32515 webserver.cc:460] Webserver started at http://127.14.127.254:39839/ using document root <none> and password file <none>
I20260923 20:26:27.909521 32515 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:27.909622 32515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:27.909973 32515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:27.913017 32515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "6abbca504e9742688cbb29c866024853"
format_stamp: "Formatted at 2026-09-23 20:26:27 on dist-test-slave-wmd0"
server_key: "d8f3edad8772af6e18ec52d9974c3e3d"
server_key_iv: "d4d839abd4dc86b22dfc2a1422c74eaf"
server_key_version: "encryptionkey@0"
I20260923 20:26:27.913842 32515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "6abbca504e9742688cbb29c866024853"
format_stamp: "Formatted at 2026-09-23 20:26:27 on dist-test-slave-wmd0"
server_key: "d8f3edad8772af6e18ec52d9974c3e3d"
server_key_iv: "d4d839abd4dc86b22dfc2a1422c74eaf"
server_key_version: "encryptionkey@0"
I20260923 20:26:27.919348 32515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260923 20:26:27.922780 32529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:27.924134 32515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:26:27.924319 32515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "6abbca504e9742688cbb29c866024853"
format_stamp: "Formatted at 2026-09-23 20:26:27 on dist-test-slave-wmd0"
server_key: "d8f3edad8772af6e18ec52d9974c3e3d"
server_key_iv: "d4d839abd4dc86b22dfc2a1422c74eaf"
server_key_version: "encryptionkey@0"
I20260923 20:26:27.924551 32515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:27.941042 32515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:27.941919 32515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:27.942229 32515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:27.961076 32515 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:44145
I20260923 20:26:27.961102 32580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:44145 every 8 connection(s)
I20260923 20:26:27.962457 32515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:27.963768 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 32515
I20260923 20:26:27.963958 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:27.964485 14847 external_mini_cluster.cc:1428] Setting key f2d9c787ad58854432c678f3bd661417
I20260923 20:26:27.967207 32581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:27.978012 32581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853: Bootstrap starting.
I20260923 20:26:27.978188 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:27.966388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.14.127.254:44145)
0923 20:26:27.967445 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:27.967456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:27.967506 (+ 50us) server_negotiation.cc:407] Connection header received
0923 20:26:27.969086 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:27.969106 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:27.969554 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:27.969988 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:27.970546 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:27.971912 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:27.973319 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:27.973983 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:27.974182 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:27.975457 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:27.975464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:27.975477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:27.976342 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:27.976457 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:27.976484 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:27.976633 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:27.976863 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 20:26:27.977220 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":376,"threads_started":1}
I20260923 20:26:27.981113 32581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:27.982082 32581 log.cc:824] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:27.984772 32581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853: No bootstrap required, opened a new log
I20260923 20:26:27.991726 32581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abbca504e9742688cbb29c866024853" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44145 } }
I20260923 20:26:27.992081 32581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:27.992208 32581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6abbca504e9742688cbb29c866024853, State: Initialized, Role: FOLLOWER
I20260923 20:26:27.992753 32581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abbca504e9742688cbb29c866024853" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44145 } }
I20260923 20:26:27.992971 32581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:27.993106 32581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:27.993300 32581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:27.995453 32581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abbca504e9742688cbb29c866024853" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44145 } }
I20260923 20:26:27.995939 32581 leader_election.cc:302] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6abbca504e9742688cbb29c866024853; no voters:
I20260923 20:26:27.996536 32581 leader_election.cc:288] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:27.996676 32586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:27.997442 32586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [term 1 LEADER]: Becoming Leader. State: Replica: 6abbca504e9742688cbb29c866024853, State: Running, Role: LEADER
I20260923 20:26:27.997954 32586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abbca504e9742688cbb29c866024853" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44145 } }
I20260923 20:26:27.998718 32581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:28.000886 32588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6abbca504e9742688cbb29c866024853. Latest consensus state: current_term: 1 leader_uuid: "6abbca504e9742688cbb29c866024853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abbca504e9742688cbb29c866024853" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44145 } } }
I20260923 20:26:28.001339 32588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:28.001685 32587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6abbca504e9742688cbb29c866024853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6abbca504e9742688cbb29c866024853" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44145 } } }
I20260923 20:26:28.002140 32587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:28.004269 32595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:28.009411 32595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:28.019268 32595 catalog_manager.cc:1347] Generated new cluster ID: 7c3a5dfdda2f4bd48dfbc60b0b7ef758
I20260923 20:26:28.019376 32595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:28.034550 32595 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:28.035856 32595 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:28.045130 32595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853: Generated new TSK 0
I20260923 20:26:28.045928 32595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:28.053763 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44145
--builtin_ntp_servers=127.14.127.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:26:28.413451 32605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:28.413761 32605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:28.413847 32605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:28.421181 32605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:28.421275 32605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:28.421422 32605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:26:28.430667 32605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:28.431932 32605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:28.433717 32605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:28.446722 32610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:28.446756 32613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:28.446954 32611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:28.448153 32605 server_base.cc:1034] running on GCE node
I20260923 20:26:28.448770 32605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:28.449749 32605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:28.450979 32605 hybrid_clock.cc:648] HybridClock initialized: now 1790195188450936 us; error 53 us; skew 500 ppm
I20260923 20:26:28.451336 32605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:28.453591 32605 webserver.cc:460] Webserver started at http://127.14.127.193:40545/ using document root <none> and password file <none>
I20260923 20:26:28.454285 32605 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:28.454419 32605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:28.454820 32605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:28.457617 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "da1271d1d5824d4088bf32333cbff878"
format_stamp: "Formatted at 2026-09-23 20:26:28 on dist-test-slave-wmd0"
server_key: "8302adf87e8e81dc072f44578e029561"
server_key_iv: "b9712459f4b58bdd8768169efc0cf732"
server_key_version: "encryptionkey@0"
I20260923 20:26:28.458439 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "da1271d1d5824d4088bf32333cbff878"
format_stamp: "Formatted at 2026-09-23 20:26:28 on dist-test-slave-wmd0"
server_key: "8302adf87e8e81dc072f44578e029561"
server_key_iv: "b9712459f4b58bdd8768169efc0cf732"
server_key_version: "encryptionkey@0"
I20260923 20:26:28.463948 32605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:26:28.467464 32619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:28.468740 32605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:26:28.468923 32605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "da1271d1d5824d4088bf32333cbff878"
format_stamp: "Formatted at 2026-09-23 20:26:28 on dist-test-slave-wmd0"
server_key: "8302adf87e8e81dc072f44578e029561"
server_key_iv: "b9712459f4b58bdd8768169efc0cf732"
server_key_version: "encryptionkey@0"
I20260923 20:26:28.469158 32605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:28.487848 32605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:28.488656 32605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:28.488970 32605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:28.489929 32605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:28.491452 32605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:28.491583 32605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:28.491704 32605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:28.491788 32605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:28.525403 32605 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:37115
I20260923 20:26:28.525457 32731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:37115 every 8 connection(s)
I20260923 20:26:28.526878 32605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:26:28.532064 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 32605
I20260923 20:26:28.532258 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:26:28.532708 14847 external_mini_cluster.cc:1428] Setting key a92887d254a4abf62d056e7da428bf4b
I20260923 20:26:28.535416 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44145
--builtin_ntp_servers=127.14.127.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:26:28.539613 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:28.529510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:58753 (local address 127.14.127.254:44145)
0923 20:26:28.529978 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:28.529984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:28.531301 (+ 1317us) server_negotiation.cc:407] Connection header received
0923 20:26:28.533166 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:28.533171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:28.533271 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:28.533390 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:28.534437 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:28.535225 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:28.536850 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:28.537250 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:28.537382 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:28.538782 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:28.538787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:28.538791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:28.538808 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:28.538847 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:28.538859 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:28.538955 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:28.539184 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:26:28.539282 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":228,"threads_started":1}
I20260923 20:26:28.540614 32735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:28.530223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:44145 (local address 127.14.127.193:58753)
0923 20:26:28.531144 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:28.531187 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:28.532905 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:28.533618 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:28.533626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:28.533716 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:28.534273 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:28.534289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:28.535384 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:28.535389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:28.536700 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:28.536712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:28.537472 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:28.537481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:28.537722 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:28.538429 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:28.538432 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:28.538471 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:28.538553 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:28.538955 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:28.538969 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:28.538979 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:26:28.539174 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 20:26:28.539571 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":256,"threads_started":1}
I20260923 20:26:28.542164 32732 heartbeater.cc:347] Connected to a master server at 127.14.127.254:44145
I20260923 20:26:28.542515 32732 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:28.543478 32732 heartbeater.cc:511] Master 127.14.127.254:44145 requested a full tablet report, sending...
I20260923 20:26:28.545918 32546 ts_manager.cc:194] Registered new tserver with Master: da1271d1d5824d4088bf32333cbff878 (127.14.127.193:37115)
I20260923 20:26:28.548163 32546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.193:58753
W20260923 20:26:28.863454 32736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:28.863742 32736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:28.863828 32736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:28.871090 32736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:28.871184 32736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:28.871285 32736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:26:28.880281 32736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:28.881582 32736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:28.883311 32736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:28.896240 32744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:28.896370 32741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:28.896347 32742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:28.897248 32736 server_base.cc:1034] running on GCE node
I20260923 20:26:28.897889 32736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:28.898876 32736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:28.900123 32736 hybrid_clock.cc:648] HybridClock initialized: now 1790195188900057 us; error 56 us; skew 500 ppm
I20260923 20:26:28.900437 32736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:28.902571 32736 webserver.cc:460] Webserver started at http://127.14.127.194:38395/ using document root <none> and password file <none>
I20260923 20:26:28.903280 32736 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:28.903419 32736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:28.903733 32736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:28.906495 32736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "aff4484e87ea4163b6e790fe6c5b29a9"
format_stamp: "Formatted at 2026-09-23 20:26:28 on dist-test-slave-wmd0"
server_key: "8aaf0573f2cb66aa0240276164f5ee9d"
server_key_iv: "50a96b54f4dd1dd6ed716de3b78d4dea"
server_key_version: "encryptionkey@0"
I20260923 20:26:28.907374 32736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "aff4484e87ea4163b6e790fe6c5b29a9"
format_stamp: "Formatted at 2026-09-23 20:26:28 on dist-test-slave-wmd0"
server_key: "8aaf0573f2cb66aa0240276164f5ee9d"
server_key_iv: "50a96b54f4dd1dd6ed716de3b78d4dea"
server_key_version: "encryptionkey@0"
I20260923 20:26:28.912312 32736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:26:28.915788 32750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:28.917032 32736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:28.917227 32736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "aff4484e87ea4163b6e790fe6c5b29a9"
format_stamp: "Formatted at 2026-09-23 20:26:28 on dist-test-slave-wmd0"
server_key: "8aaf0573f2cb66aa0240276164f5ee9d"
server_key_iv: "50a96b54f4dd1dd6ed716de3b78d4dea"
server_key_version: "encryptionkey@0"
I20260923 20:26:28.917445 32736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:28.931599 32736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:28.932406 32736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:28.932708 32736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:28.933653 32736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:28.935081 32736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:28.935217 32736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:28.935351 32736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:28.935446 32736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:28.968848 32736 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:42689
I20260923 20:26:28.968956 394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:42689 every 8 connection(s)
I20260923 20:26:28.970185 32736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:26:28.972182 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 32736
I20260923 20:26:28.972375 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:26:28.972985 14847 external_mini_cluster.cc:1428] Setting key a0852f59d8e14c80286a0d4b4edfc4b7
I20260923 20:26:28.975975 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44145
--builtin_ntp_servers=127.14.127.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:26:28.984803 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:28.972982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:52073 (local address 127.14.127.254:44145)
0923 20:26:28.973181 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:28.973187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:28.974904 (+ 1717us) server_negotiation.cc:407] Connection header received
0923 20:26:28.977446 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:28.977454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:28.977544 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:28.977655 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:28.978939 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:28.980124 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:28.981625 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:28.982344 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:28.982491 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:28.983896 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:28.983904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:28.983910 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:28.983934 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:28.983976 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:28.983993 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:28.984095 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:28.984306 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 20:26:28.984417 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:26:28.985543 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:28.973650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:44145 (local address 127.14.127.194:52073)
0923 20:26:28.974760 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:28.974810 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:28.977119 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:28.977892 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:28.977904 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:28.978014 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:28.978764 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:28.978785 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:28.980271 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:28.980275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:28.981473 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:28.981485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:28.982542 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:28.982547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:28.982818 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:28.983583 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:28.983586 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:28.983626 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:28.983699 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:28.984090 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:28.984098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:28.984104 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:26:28.984238 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:26:28.984514 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":357,"threads_started":1}
I20260923 20:26:28.987040 395 heartbeater.cc:347] Connected to a master server at 127.14.127.254:44145
I20260923 20:26:28.987337 395 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:28.988031 395 heartbeater.cc:511] Master 127.14.127.254:44145 requested a full tablet report, sending...
I20260923 20:26:28.989423 32546 ts_manager.cc:194] Registered new tserver with Master: aff4484e87ea4163b6e790fe6c5b29a9 (127.14.127.194:42689)
I20260923 20:26:28.990211 32546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.194:52073
W20260923 20:26:29.301806 398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:29.302091 398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:29.302232 398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:29.309751 398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:29.309847 398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:29.310050 398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:26:29.319231 398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:29.320603 398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:29.322386 398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:29.335551 404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:29.335618 406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:29.335593 403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:29.336185 398 server_base.cc:1034] running on GCE node
I20260923 20:26:29.336833 398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:29.337738 398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:29.338972 398 hybrid_clock.cc:648] HybridClock initialized: now 1790195189338916 us; error 48 us; skew 500 ppm
I20260923 20:26:29.339305 398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:29.341493 398 webserver.cc:460] Webserver started at http://127.14.127.195:35991/ using document root <none> and password file <none>
I20260923 20:26:29.342192 398 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:29.342321 398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:29.342664 398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:29.345384 398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "ef7bed3a8e53429591c903f3513a0302"
format_stamp: "Formatted at 2026-09-23 20:26:29 on dist-test-slave-wmd0"
server_key: "48b4b920d16997e9a40301a03486343e"
server_key_iv: "3d8f2cdc63c4e1871b5d13cf60c9718d"
server_key_version: "encryptionkey@0"
I20260923 20:26:29.346133 398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "ef7bed3a8e53429591c903f3513a0302"
format_stamp: "Formatted at 2026-09-23 20:26:29 on dist-test-slave-wmd0"
server_key: "48b4b920d16997e9a40301a03486343e"
server_key_iv: "3d8f2cdc63c4e1871b5d13cf60c9718d"
server_key_version: "encryptionkey@0"
I20260923 20:26:29.351246 398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 20:26:29.354499 412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:29.355856 398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:26:29.356014 398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "ef7bed3a8e53429591c903f3513a0302"
format_stamp: "Formatted at 2026-09-23 20:26:29 on dist-test-slave-wmd0"
server_key: "48b4b920d16997e9a40301a03486343e"
server_key_iv: "3d8f2cdc63c4e1871b5d13cf60c9718d"
server_key_version: "encryptionkey@0"
I20260923 20:26:29.356225 398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:29.383201 398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:29.384043 398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:29.384351 398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:29.385324 398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:29.386826 398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:29.386961 398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:29.387095 398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:29.387195 398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:29.419763 398 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:43663
I20260923 20:26:29.419850 524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:43663 every 8 connection(s)
I20260923 20:26:29.421119 398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:26:29.422041 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 398
I20260923 20:26:29.422220 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:26:29.422837 14847 external_mini_cluster.cc:1428] Setting key 629e930afb43bdc38e292b8a1eac1e14
I20260923 20:26:29.433611 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.424162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:56773 (local address 127.14.127.254:44145)
0923 20:26:29.424328 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.424333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.425943 (+ 1610us) server_negotiation.cc:407] Connection header received
0923 20:26:29.427883 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.427887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.427950 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.428029 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:29.429114 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.429834 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.431330 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.431734 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.431824 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.433001 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:29.433005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:29.433009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:29.433025 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:29.433055 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:29.433067 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:29.433133 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.433314 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.433385 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 20:26:29.434569 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.424664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:44145 (local address 127.14.127.195:56773)
0923 20:26:29.425754 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.425816 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:29.427625 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:29.428265 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:29.428274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:29.428363 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:29.428975 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.428991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.429950 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.429954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.431210 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.431222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.431907 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.431911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.432119 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.432725 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:29.432727 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:29.432765 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:29.432840 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:29.433144 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:29.433151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:29.433156 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:26:29.433286 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 20:26:29.433550 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":278,"threads_started":1}
I20260923 20:26:29.435909 525 heartbeater.cc:347] Connected to a master server at 127.14.127.254:44145
I20260923 20:26:29.436195 525 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:29.437104 525 heartbeater.cc:511] Master 127.14.127.254:44145 requested a full tablet report, sending...
I20260923 20:26:29.438553 32546 ts_manager.cc:194] Registered new tserver with Master: ef7bed3a8e53429591c903f3513a0302 (127.14.127.195:43663)
I20260923 20:26:29.439352 32546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.195:56773
I20260923 20:26:29.450191 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:26:29.459959 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.454201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.14.127.254:44145)
0923 20:26:29.454475 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.454481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.454823 (+ 342us) server_negotiation.cc:407] Connection header received
0923 20:26:29.454980 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.454986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.455072 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.455166 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:29.455585 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.456319 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.457701 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.458123 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.458227 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.459310 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:29.459315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:29.459318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:29.459352 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:29.459383 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:29.459395 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:29.459462 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.459569 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.459643 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:26:29.463961 32546 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:26:29.470783 32546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:26:29.494340 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.483377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.14.127.194:42689)
0923 20:26:29.483975 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.483984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.484571 (+ 587us) server_negotiation.cc:407] Connection header received
0923 20:26:29.484920 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.484932 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.485207 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.485386 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:29.487080 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.488471 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.491290 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.492789 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.492924 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.493138 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.493801 (+ 663us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.493973 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":328,"threads_started":1}
I20260923 20:26:29.494453 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.482408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46584)
0923 20:26:29.483206 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.483250 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:29.483423 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:29.487023 (+ 3600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:29.487028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:29.487053 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:29.487392 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.487404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.489579 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.489583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.491629 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.491644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.493442 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.493448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.493619 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.493662 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 20:26:29.493755 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 20:26:29.494008 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":48,"thread_start_us":342,"threads_started":1}
I20260923 20:26:29.494951 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.483028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:42689 (local address 127.0.0.1:60580)
0923 20:26:29.484447 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.484469 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:29.484710 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:29.485794 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:29.485806 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:29.485848 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:29.486821 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.486835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.488621 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.488627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.491050 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.491062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.493318 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.493327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.493537 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.493588 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:26:29.493750 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 20:26:29.494006 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"mutex_wait_us":52,"thread_start_us":3338,"threads_started":1}
I20260923 20:26:29.494999 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.482596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.14.127.193:37115)
0923 20:26:29.486409 (+ 3813us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.486418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.486448 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:26:29.486542 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.486551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.486821 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.487041 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:29.488254 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.489397 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.491849 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.493237 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.493395 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.493594 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.494427 (+ 833us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.494610 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3636,"thread_start_us":3534,"threads_started":1}
I20260923 20:26:29.495877 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.484234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.14.127.195:43663)
0923 20:26:29.485761 (+ 1527us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.485774 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.486824 (+ 1050us) server_negotiation.cc:407] Connection header received
0923 20:26:29.487582 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.487589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.487776 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.487961 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:29.488787 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.490313 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.492707 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.493791 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.493968 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.494178 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.495365 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.495520 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":1125,"threads_started":1}
I20260923 20:26:29.496500 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.483629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:43663 (local address 127.0.0.1:58970)
0923 20:26:29.486711 (+ 3082us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.486726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:29.486872 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:29.488158 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:29.488166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:29.488195 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:29.488572 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.488587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.490462 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.490467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.492528 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.492540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.494380 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.494393 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.494755 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.494790 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:26:29.494876 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:26:29.496269 (+ 1393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2904,"mutex_wait_us":1,"thread_start_us":519,"threads_started":1}
I20260923 20:26:29.498867 330 tablet_service.cc:1467] Processing CreateTablet for tablet 837a2309a19048a7aaa072aafcb631fc (DEFAULT_TABLE table=test-table [id=4e67353d248842859b2297c8a82e6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:29.498867 32667 tablet_service.cc:1467] Processing CreateTablet for tablet 837a2309a19048a7aaa072aafcb631fc (DEFAULT_TABLE table=test-table [id=4e67353d248842859b2297c8a82e6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:29.500316 330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837a2309a19048a7aaa072aafcb631fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:29.500319 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837a2309a19048a7aaa072aafcb631fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:29.501454 460 tablet_service.cc:1467] Processing CreateTablet for tablet 837a2309a19048a7aaa072aafcb631fc (DEFAULT_TABLE table=test-table [id=4e67353d248842859b2297c8a82e6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:29.502683 460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837a2309a19048a7aaa072aafcb631fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:29.515439 543 tablet_bootstrap.cc:492] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Bootstrap starting.
I20260923 20:26:29.516535 544 tablet_bootstrap.cc:492] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: Bootstrap starting.
I20260923 20:26:29.519388 545 tablet_bootstrap.cc:492] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878: Bootstrap starting.
I20260923 20:26:29.520373 544 tablet_bootstrap.cc:654] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:29.520380 543 tablet_bootstrap.cc:654] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:29.521693 544 log.cc:824] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:29.521700 543 log.cc:824] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:29.522913 545 tablet_bootstrap.cc:654] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:29.524224 545 log.cc:824] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:29.525305 544 tablet_bootstrap.cc:492] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: No bootstrap required, opened a new log
I20260923 20:26:29.525601 544 ts_tablet_manager.cc:1397] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260923 20:26:29.525861 543 tablet_bootstrap.cc:492] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: No bootstrap required, opened a new log
I20260923 20:26:29.526153 543 ts_tablet_manager.cc:1397] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260923 20:26:29.533537 545 tablet_bootstrap.cc:492] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878: No bootstrap required, opened a new log
I20260923 20:26:29.533823 545 ts_tablet_manager.cc:1397] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260923 20:26:29.536222 544 raft_consensus.cc:348] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.536705 544 raft_consensus.cc:374] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:29.536846 544 raft_consensus.cc:729] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff4484e87ea4163b6e790fe6c5b29a9, State: Initialized, Role: FOLLOWER
I20260923 20:26:29.537516 544 consensus_queue.cc:260] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.538627 543 raft_consensus.cc:348] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.539096 543 raft_consensus.cc:374] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:29.539232 543 raft_consensus.cc:729] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7bed3a8e53429591c903f3513a0302, State: Initialized, Role: FOLLOWER
I20260923 20:26:29.539918 543 consensus_queue.cc:260] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.544777 395 heartbeater.cc:503] Master 127.14.127.254:44145 was elected leader, sending a full tablet report...
I20260923 20:26:29.545225 525 heartbeater.cc:503] Master 127.14.127.254:44145 was elected leader, sending a full tablet report...
I20260923 20:26:29.546061 543 ts_tablet_manager.cc:1428] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260923 20:26:29.546886 544 ts_tablet_manager.cc:1428] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: Time spent starting tablet: real 0.021s user 0.017s sys 0.005s
I20260923 20:26:29.547422 545 raft_consensus.cc:348] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.547892 545 raft_consensus.cc:374] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:29.547998 545 raft_consensus.cc:729] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1271d1d5824d4088bf32333cbff878, State: Initialized, Role: FOLLOWER
I20260923 20:26:29.548575 545 consensus_queue.cc:260] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.549882 32732 heartbeater.cc:503] Master 127.14.127.254:44145 was elected leader, sending a full tablet report...
I20260923 20:26:29.550637 545 ts_tablet_manager.cc:1428] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
I20260923 20:26:29.636917 550 raft_consensus.cc:482] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:26:29.637284 550 raft_consensus.cc:504] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.639060 550 leader_election.cc:288] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da1271d1d5824d4088bf32333cbff878 (127.14.127.193:37115), aff4484e87ea4163b6e790fe6c5b29a9 (127.14.127.194:42689)
I20260923 20:26:29.646660 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.639494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:37621 (local address 127.14.127.193:37115)
0923 20:26:29.639723 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.639729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.640004 (+ 275us) server_negotiation.cc:407] Connection header received
0923 20:26:29.640167 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.640175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.640267 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.640351 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:29.640872 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.641832 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.644407 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.645623 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.645806 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.645888 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.646234 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.646358 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:26:29.646658 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.639722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:51157 (local address 127.14.127.194:42689)
0923 20:26:29.639939 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.639948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.640473 (+ 525us) server_negotiation.cc:407] Connection header received
0923 20:26:29.640580 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.640584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.640638 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.640705 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:29.641244 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.642147 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.644565 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.645763 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.645898 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.645965 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.646234 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.646359 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:26:29.646689 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.639445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.14.127.195:37621)
0923 20:26:29.639796 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.639818 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:29.639994 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:29.640419 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:29.640427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:29.640456 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:29.640675 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.640711 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.641961 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.641964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.644216 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.644233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.645788 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.645792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.646013 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.646087 (+ 74us) client_negotiation.cc:769] Sending connection context
0923 20:26:29.646186 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 20:26:29.646334 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":113}
I20260923 20:26:29.646742 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.639594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:42689 (local address 127.14.127.195:51157)
0923 20:26:29.640354 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.640373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:29.640476 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:29.640751 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:29.640757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:29.640786 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:29.641089 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.641098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.642286 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.642291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.644395 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:29.644408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.645921 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.645927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:29.646083 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.646107 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:26:29.646190 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 20:26:29.646334 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":3,"thread_start_us":396,"threads_started":1}
I20260923 20:26:29.647863 32687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837a2309a19048a7aaa072aafcb631fc" candidate_uuid: "ef7bed3a8e53429591c903f3513a0302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1271d1d5824d4088bf32333cbff878" is_pre_election: true
I20260923 20:26:29.647863 350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837a2309a19048a7aaa072aafcb631fc" candidate_uuid: "ef7bed3a8e53429591c903f3513a0302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" is_pre_election: true
I20260923 20:26:29.648470 32687 raft_consensus.cc:2454] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef7bed3a8e53429591c903f3513a0302 in term 0.
I20260923 20:26:29.648466 350 raft_consensus.cc:2454] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef7bed3a8e53429591c903f3513a0302 in term 0.
I20260923 20:26:29.649245 413 leader_election.cc:302] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: aff4484e87ea4163b6e790fe6c5b29a9, da1271d1d5824d4088bf32333cbff878, ef7bed3a8e53429591c903f3513a0302; no voters:
I20260923 20:26:29.649663 550 raft_consensus.cc:2781] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:26:29.649802 550 raft_consensus.cc:482] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:26:29.649924 550 raft_consensus.cc:3037] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:29.652305 550 raft_consensus.cc:504] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.652940 550 leader_election.cc:288] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [CANDIDATE]: Term 1 election: Requested vote from peers da1271d1d5824d4088bf32333cbff878 (127.14.127.193:37115), aff4484e87ea4163b6e790fe6c5b29a9 (127.14.127.194:42689)
I20260923 20:26:29.653317 32687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837a2309a19048a7aaa072aafcb631fc" candidate_uuid: "ef7bed3a8e53429591c903f3513a0302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1271d1d5824d4088bf32333cbff878"
I20260923 20:26:29.653430 350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837a2309a19048a7aaa072aafcb631fc" candidate_uuid: "ef7bed3a8e53429591c903f3513a0302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff4484e87ea4163b6e790fe6c5b29a9"
I20260923 20:26:29.653523 32687 raft_consensus.cc:3037] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:29.653669 350 raft_consensus.cc:3037] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:29.655866 32687 raft_consensus.cc:2454] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef7bed3a8e53429591c903f3513a0302 in term 1.
I20260923 20:26:29.656020 350 raft_consensus.cc:2454] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef7bed3a8e53429591c903f3513a0302 in term 1.
I20260923 20:26:29.656358 414 leader_election.cc:302] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da1271d1d5824d4088bf32333cbff878, ef7bed3a8e53429591c903f3513a0302; no voters:
I20260923 20:26:29.656706 550 raft_consensus.cc:2781] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:29.657153 550 raft_consensus.cc:686] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 1 LEADER]: Becoming Leader. State: Replica: ef7bed3a8e53429591c903f3513a0302, State: Running, Role: LEADER
I20260923 20:26:29.657617 550 consensus_queue.cc:237] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } }
I20260923 20:26:29.662003 32544 catalog_manager.cc:5495] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef7bed3a8e53429591c903f3513a0302 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "ef7bed3a8e53429591c903f3513a0302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef7bed3a8e53429591c903f3513a0302" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 43663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 } health_report { overall_health: UNKNOWN } } }
W20260923 20:26:29.673013 526 tablet.cc:2369] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:26:29.685889 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.680956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.14.127.195:43663)
0923 20:26:29.681191 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.681195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.681213 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:26:29.681408 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.681412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.681486 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.681605 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:26:29.681943 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.682698 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.683975 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.684355 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.684440 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.684884 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:26:29.685346 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:26:29.685457 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.685533 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.685592 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:26:29.694021 350 raft_consensus.cc:1261] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 1 FOLLOWER]: Refusing update from remote peer ef7bed3a8e53429591c903f3513a0302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:29.694025 32687 raft_consensus.cc:1261] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 1 FOLLOWER]: Refusing update from remote peer ef7bed3a8e53429591c903f3513a0302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:29.694993 550 consensus_queue.cc:1035] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff4484e87ea4163b6e790fe6c5b29a9" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 42689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:29.695323 553 consensus_queue.cc:1035] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1271d1d5824d4088bf32333cbff878" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 37115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:29.711817 557 mvcc.cc:205] Tried to move back new op lower bound from 7332639496976490496 to 7332639496841199616. Current Snapshot: MvccSnapshot[applied={T|T < 7332639496976490496}]
I20260923 20:26:29.721611 32545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:29.721879 32545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:29.725400 32545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6abbca504e9742688cbb29c866024853: Sending DeleteTablet for 3 replicas of tablet 837a2309a19048a7aaa072aafcb631fc
I20260923 20:26:29.726657 330 tablet_service.cc:1514] Processing DeleteTablet for tablet 837a2309a19048a7aaa072aafcb631fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:29 UTC) from {username='slave'} at 127.0.0.1:60580
I20260923 20:26:29.726799 460 tablet_service.cc:1514] Processing DeleteTablet for tablet 837a2309a19048a7aaa072aafcb631fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:29 UTC) from {username='slave'} at 127.0.0.1:58970
I20260923 20:26:29.727453 32667 tablet_service.cc:1514] Processing DeleteTablet for tablet 837a2309a19048a7aaa072aafcb631fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:29 UTC) from {username='slave'} at 127.0.0.1:46584
I20260923 20:26:29.727677 566 tablet_replica.cc:330] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: stopping tablet replica
I20260923 20:26:29.728048 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 32605
I20260923 20:26:29.728245 566 raft_consensus.cc:2229] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:29.729028 566 raft_consensus.cc:2258] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:29.732635 566 ts_tablet_manager.cc:1905] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:29.735169 568 tablet_replica.cc:330] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878: stopping tablet replica
I20260923 20:26:29.735590 568 raft_consensus.cc:2229] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:29.736148 568 raft_consensus.cc:2258] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:29.736930 567 tablet_replica.cc:330] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: stopping tablet replica
I20260923 20:26:29.737298 567 raft_consensus.cc:2229] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:26:29.737936 567 raft_consensus.cc:2258] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:29.738235 568 ts_tablet_manager.cc:1905] T 837a2309a19048a7aaa072aafcb631fc P da1271d1d5824d4088bf32333cbff878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:29.739933 567 ts_tablet_manager.cc:1905] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:29.742978 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.730219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.14.127.193:37115)
0923 20:26:29.730437 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.730446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.730992 (+ 546us) server_negotiation.cc:407] Connection header received
0923 20:26:29.734971 (+ 3979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.734975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.735070 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.735135 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:29.735687 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.736923 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.739071 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.739744 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.739848 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.741336 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:29.741346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:29.741362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:29.742063 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:29.742127 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:29.742150 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:29.742236 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.742489 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.742604 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:26:29.744218 32727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:29.747243 567 ts_tablet_manager.cc:1918] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:29.747418 567 log.cc:1196] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/837a2309a19048a7aaa072aafcb631fc
I20260923 20:26:29.747964 567 ts_tablet_manager.cc:1939] T 837a2309a19048a7aaa072aafcb631fc P ef7bed3a8e53429591c903f3513a0302: Deleting consensus metadata
I20260923 20:26:29.748468 566 ts_tablet_manager.cc:1918] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:29.748646 566 log.cc:1196] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/837a2309a19048a7aaa072aafcb631fc
I20260923 20:26:29.749293 566 ts_tablet_manager.cc:1939] T 837a2309a19048a7aaa072aafcb631fc P aff4484e87ea4163b6e790fe6c5b29a9: Deleting consensus metadata
I20260923 20:26:29.749811 32531 catalog_manager.cc:4841] TS ef7bed3a8e53429591c903f3513a0302 (127.14.127.195:43663): tablet 837a2309a19048a7aaa072aafcb631fc (table test-table [id=4e67353d248842859b2297c8a82e6220]) successfully deleted
I20260923 20:26:29.751329 32530 catalog_manager.cc:4841] TS aff4484e87ea4163b6e790fe6c5b29a9 (127.14.127.194:42689): tablet 837a2309a19048a7aaa072aafcb631fc (table test-table [id=4e67353d248842859b2297c8a82e6220]) successfully deleted
I20260923 20:26:29.839967 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 32605
I20260923 20:26:29.858390 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 32736
W20260923 20:26:29.858407 32531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.127.193:37115 (error 108)
I20260923 20:26:29.860905 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.860392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46608)
0923 20:26:29.860566 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.860722 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260923 20:26:29.861794 32531 catalog_manager.cc:4584] TS da1271d1d5824d4088bf32333cbff878 (127.14.127.193:37115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 837a2309a19048a7aaa072aafcb631fc: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
I20260923 20:26:29.866398 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.859600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.14.127.194:42689)
0923 20:26:29.859796 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.859810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.859877 (+ 67us) server_negotiation.cc:407] Connection header received
0923 20:26:29.859944 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.859949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.860055 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.860139 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:29.860683 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.861575 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.863495 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.863881 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.863976 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.865069 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:29.865077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:29.865091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:29.865648 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:29.865695 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:29.865720 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:29.865808 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.865973 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.866055 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:26:29.867129 390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:29.915529 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.914956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46614)
0923 20:26:29.915215 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.915328 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 20:26:29.916734 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.916326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46620)
0923 20:26:29.916485 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.916585 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 20:26:29.955727 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 32736
I20260923 20:26:29.977265 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.976820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46628)
0923 20:26:29.977025 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.977105 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:26:29.978670 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.978012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46644)
0923 20:26:29.978418 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:29.978517 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314}
I20260923 20:26:29.978667 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 398
I20260923 20:26:29.997064 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:29.983058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.14.127.195:43663)
0923 20:26:29.983271 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:29.983277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:29.983297 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:26:29.983432 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:29.983442 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:29.983550 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:29.983676 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:29.984089 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.984899 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.986191 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:29.986828 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:29.986980 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:29.995620 (+ 8640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:29.995630 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:29.995643 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:29.996261 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:29.996311 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:29.996337 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:29.996468 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:29.996609 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 20:26:29.996712 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:26:29.997747 520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:30.059487 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:30.058975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46654)
0923 20:26:30.059198 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:30.059336 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:26:30.060557 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:30.060185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46670)
0923 20:26:30.060384 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:30.060452 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 20:26:30.119822 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 398
I20260923 20:26:30.139452 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 32515
I20260923 20:26:30.140550 32576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:30.248829 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:30.248261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:37115 (local address 127.0.0.1:46682)
0923 20:26:30.248508 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:30.248608 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 20:26:30.248831 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 32515
2026-09-23T20:26:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2783 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:30 dist-test-slave-wmd0 krb5kdc[577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:30 dist-test-slave-wmd0 krb5kdc[577](info): set up 2 sockets
Sep 23 20:26:30 dist-test-slave-wmd0 krb5kdc[577](info): commencing operation
krb5kdc: starting...
W20260923 20:26:31.366905 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:31 dist-test-slave-wmd0 krb5kdc[577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:26:31.609247 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46347
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:46347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:31.939618 590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:31.939914 590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:31.940116 590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:31.947474 590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:31.947587 590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:31.947690 590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:31.947789 590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:31.947852 590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:31.957981 590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:46347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:31.959424 590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:31.960767 590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:31.973201 598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:31.973248 596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:31.973299 595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:31.976317 590 server_base.cc:1034] running on GCE node
I20260923 20:26:31.977229 590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:31.978924 590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:31.980171 590 hybrid_clock.cc:648] HybridClock initialized: now 1790195191980131 us; error 61 us; skew 500 ppm
I20260923 20:26:31.980548 590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:31 dist-test-slave-wmd0 krb5kdc[577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195191, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:31.983958 590 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:26:31.985635 590 webserver.cc:460] Webserver started at http://127.14.127.254:39829/ using document root <none> and password file <none>
I20260923 20:26:31.986393 590 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:31.986545 590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:31.986912 590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:31.990065 590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "8a72f178750e43929db4ce45c8da5f9f"
format_stamp: "Formatted at 2026-09-23 20:26:31 on dist-test-slave-wmd0"
server_key: "21de8fe24c19ebbf2e667c232080bf3e"
server_key_iv: "39b92181f7d9fbd7b8683faf9194fb16"
server_key_version: "encryptionkey@0"
I20260923 20:26:31.990929 590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "8a72f178750e43929db4ce45c8da5f9f"
format_stamp: "Formatted at 2026-09-23 20:26:31 on dist-test-slave-wmd0"
server_key: "21de8fe24c19ebbf2e667c232080bf3e"
server_key_iv: "39b92181f7d9fbd7b8683faf9194fb16"
server_key_version: "encryptionkey@0"
I20260923 20:26:31.996337 590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:26:31.999810 605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:32.001118 590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 20:26:32.001329 590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "8a72f178750e43929db4ce45c8da5f9f"
format_stamp: "Formatted at 2026-09-23 20:26:31 on dist-test-slave-wmd0"
server_key: "21de8fe24c19ebbf2e667c232080bf3e"
server_key_iv: "39b92181f7d9fbd7b8683faf9194fb16"
server_key_version: "encryptionkey@0"
I20260923 20:26:32.001545 590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:32.024354 590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:32.029013 590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:32.029273 590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:32.047816 590 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:46347
I20260923 20:26:32.047834 656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:46347 every 8 connection(s)
I20260923 20:26:32.049317 590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:32.053172 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:32.053323 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 590
I20260923 20:26:32.053493 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:32.054014 14847 external_mini_cluster.cc:1428] Setting key 0bf4a5c86633c195044c56090aaa9514
I20260923 20:26:32.068419 657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f: Bootstrap starting.
Sep 23 20:26:32 dist-test-slave-wmd0 krb5kdc[577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:32.071905 657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:32.072996 657 log.cc:824] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:32.075845 657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f: No bootstrap required, opened a new log
I20260923 20:26:32.077597 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:32.055971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.14.127.254:46347)
0923 20:26:32.056847 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:32.056859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:32.056903 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:26:32.058548 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:32.058569 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:32.059050 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:32.059400 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:32.061169 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:32.062487 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:32.064377 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:32.065129 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:32.065268 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:32.070222 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:32.070231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:32.070247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:32.070292 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:32.073471 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:32.074205 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:32.074211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:32.074218 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:32.074391 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:32.074857 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:32.074863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:32.074868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:32.075399 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:32.075727 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:32.076173 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 20:26:32.076616 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":454,"threads_started":1}
I20260923 20:26:32.084230 657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a72f178750e43929db4ce45c8da5f9f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46347 } }
I20260923 20:26:32.084604 657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:32.084717 657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a72f178750e43929db4ce45c8da5f9f, State: Initialized, Role: FOLLOWER
I20260923 20:26:32.085295 657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a72f178750e43929db4ce45c8da5f9f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46347 } }
I20260923 20:26:32.085533 657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:32.085687 657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:32.085860 657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:32.088425 657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a72f178750e43929db4ce45c8da5f9f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46347 } }
I20260923 20:26:32.088932 657 leader_election.cc:302] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a72f178750e43929db4ce45c8da5f9f; no voters:
I20260923 20:26:32.089579 657 leader_election.cc:288] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:32.089702 662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:32.090828 662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [term 1 LEADER]: Becoming Leader. State: Replica: 8a72f178750e43929db4ce45c8da5f9f, State: Running, Role: LEADER
I20260923 20:26:32.091331 662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a72f178750e43929db4ce45c8da5f9f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46347 } }
I20260923 20:26:32.091738 657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:32.094288 663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a72f178750e43929db4ce45c8da5f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a72f178750e43929db4ce45c8da5f9f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46347 } } }
I20260923 20:26:32.094265 664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a72f178750e43929db4ce45c8da5f9f. Latest consensus state: current_term: 1 leader_uuid: "8a72f178750e43929db4ce45c8da5f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a72f178750e43929db4ce45c8da5f9f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46347 } } }
I20260923 20:26:32.094799 664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:32.094799 663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:32.095726 669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:32.101599 669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:32.111541 669 catalog_manager.cc:1347] Generated new cluster ID: de9398aaf2d244c8a2389e0a0e065fb6
I20260923 20:26:32.111689 669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:32.135816 669 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:32.137204 669 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:32.143976 669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f: Generated new TSK 0
I20260923 20:26:32.144827 669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:26:32.206512 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46347
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:32.537688 683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:32.538033 683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:32.538213 683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:32.545522 683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:32.545689 683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:32.545888 683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:26:32.555586 683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:32.556881 683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:32.558200 683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:32.571262 688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:32.571233 689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:32.571216 691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:32.572078 683 server_base.cc:1034] running on GCE node
I20260923 20:26:32.572693 683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:32.573630 683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:32.574842 683 hybrid_clock.cc:648] HybridClock initialized: now 1790195192574790 us; error 54 us; skew 500 ppm
I20260923 20:26:32.575165 683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:32 dist-test-slave-wmd0 krb5kdc[577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195192, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:32.578110 683 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:26:32.579699 683 webserver.cc:460] Webserver started at http://127.14.127.193:42451/ using document root <none> and password file <none>
I20260923 20:26:32.580372 683 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:32.580518 683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:32.580830 683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:32.583808 683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "85ad8565bd144f25b234dc6101d0b727"
format_stamp: "Formatted at 2026-09-23 20:26:32 on dist-test-slave-wmd0"
server_key: "99bacfacb1183f316a1e4531c57aece9"
server_key_iv: "813aa93f97e50c8f15b889d60b375d76"
server_key_version: "encryptionkey@0"
I20260923 20:26:32.584573 683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "85ad8565bd144f25b234dc6101d0b727"
format_stamp: "Formatted at 2026-09-23 20:26:32 on dist-test-slave-wmd0"
server_key: "99bacfacb1183f316a1e4531c57aece9"
server_key_iv: "813aa93f97e50c8f15b889d60b375d76"
server_key_version: "encryptionkey@0"
I20260923 20:26:32.589637 683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:26:32.593230 698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:32.594523 683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:26:32.594745 683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "85ad8565bd144f25b234dc6101d0b727"
format_stamp: "Formatted at 2026-09-23 20:26:32 on dist-test-slave-wmd0"
server_key: "99bacfacb1183f316a1e4531c57aece9"
server_key_iv: "813aa93f97e50c8f15b889d60b375d76"
server_key_version: "encryptionkey@0"
I20260923 20:26:32.594978 683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:32.611145 683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:32.614987 683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:32.615233 683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:32.616115 683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:32.617367 683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:32.617439 683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:32.617523 683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:32.617566 683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:32.649371 683 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:44783
I20260923 20:26:32.649408 810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:44783 every 8 connection(s)
I20260923 20:26:32.650738 683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:26:32.653288 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 683
I20260923 20:26:32.653477 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:26:32.653913 14847 external_mini_cluster.cc:1428] Setting key b390e5869b32151b40346f1bef50c6c3
Sep 23 20:26:32 dist-test-slave-wmd0 krb5kdc[577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195192, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:32.670926 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:32.653293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:51895 (local address 127.14.127.254:46347)
0923 20:26:32.656125 (+ 2832us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:32.656130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:32.656148 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:26:32.657019 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:32.657025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:32.657111 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:32.657225 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:32.658837 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:32.659542 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:32.661053 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:32.661452 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:32.661582 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:32.665247 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:32.665252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:32.665256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:32.665288 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:32.667865 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:32.668613 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:32.668617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:32.668620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:32.668737 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:32.669185 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:32.669195 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:32.669197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:32.669448 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:32.669615 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:32.670312 (+ 697us) server_negotiation.cc:299] Negotiation successful
0923 20:26:32.670531 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2700,"thread_start_us":2598,"threads_started":1}
I20260923 20:26:32.671870 814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:32.654025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46347 (local address 127.14.127.193:51895)
0923 20:26:32.655092 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:32.655131 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:32.656756 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:32.657465 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:32.657473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:32.658014 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:32.658648 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:32.658674 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:32.659702 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:32.659707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:32.660908 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:32.660919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:32.661682 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:32.661691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:32.661927 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:32.662705 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:32.662727 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:32.665023 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:32.668015 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:32.668025 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:32.668040 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:32.668467 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:32.668854 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:32.668859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:32.668862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:32.669054 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:32.669583 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:32.669591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:32.669927 (+ 336us) client_negotiation.cc:769] Sending connection context
0923 20:26:32.670200 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 20:26:32.670732 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":573,"threads_started":1}
I20260923 20:26:32.673456 811 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46347
I20260923 20:26:32.673794 811 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:32.674713 811 heartbeater.cc:511] Master 127.14.127.254:46347 requested a full tablet report, sending...
I20260923 20:26:32.677220 622 ts_manager.cc:194] Registered new tserver with Master: 85ad8565bd144f25b234dc6101d0b727 (127.14.127.193:44783)
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
I20260923 20:26:32.679397 622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:51895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:26:32.709717 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46347
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:33.032303 817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:33.032568 817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:33.032742 817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:33.039944 817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:33.040086 817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:33.040277 817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:26:33.051832 817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:33.053423 817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:33.055035 817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:33.069588 822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:33.069821 825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:33.070171 823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:33.070597 817 server_base.cc:1034] running on GCE node
I20260923 20:26:33.071393 817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:33.072674 817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:33.073918 817 hybrid_clock.cc:648] HybridClock initialized: now 1790195193073849 us; error 56 us; skew 500 ppm
I20260923 20:26:33.074323 817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:33 dist-test-slave-wmd0 krb5kdc[577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195193, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:33.077622 817 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:26:33.079111 817 webserver.cc:460] Webserver started at http://127.14.127.194:36567/ using document root <none> and password file <none>
I20260923 20:26:33.079789 817 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:33.079926 817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:33.080245 817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:33.083112 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "c5f09bd1b73a4242807c8f87b33ad7bf"
format_stamp: "Formatted at 2026-09-23 20:26:33 on dist-test-slave-wmd0"
server_key: "93b505bef770261fde09f33d4f6e79b0"
server_key_iv: "5af011366a43adf315f582a24a12b9d4"
server_key_version: "encryptionkey@0"
I20260923 20:26:33.083865 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "c5f09bd1b73a4242807c8f87b33ad7bf"
format_stamp: "Formatted at 2026-09-23 20:26:33 on dist-test-slave-wmd0"
server_key: "93b505bef770261fde09f33d4f6e79b0"
server_key_iv: "5af011366a43adf315f582a24a12b9d4"
server_key_version: "encryptionkey@0"
I20260923 20:26:33.088840 817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 20:26:33.092283 832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:33.093647 817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:26:33.093812 817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "c5f09bd1b73a4242807c8f87b33ad7bf"
format_stamp: "Formatted at 2026-09-23 20:26:33 on dist-test-slave-wmd0"
server_key: "93b505bef770261fde09f33d4f6e79b0"
server_key_iv: "5af011366a43adf315f582a24a12b9d4"
server_key_version: "encryptionkey@0"
I20260923 20:26:33.094009 817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:33.130640 817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:33.134590 817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:33.134899 817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:33.135872 817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:33.137157 817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:33.137267 817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:33.137411 817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:33.137482 817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:33.170387 817 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:39021
I20260923 20:26:33.170447 944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:39021 every 8 connection(s)
I20260923 20:26:33.171722 817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:26:33.173866 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 817
I20260923 20:26:33.174057 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:26:33.174647 14847 external_mini_cluster.cc:1428] Setting key b99f2f94dd5a0c35f423d9176544539a
Sep 23 20:26:33 dist-test-slave-wmd0 krb5kdc[577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195193, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:33.195338 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.174362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:43331 (local address 127.14.127.254:46347)
0923 20:26:33.174848 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:33.174852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:33.176029 (+ 1177us) server_negotiation.cc:407] Connection header received
0923 20:26:33.178531 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:33.178535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:33.178597 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:33.178738 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:33.180774 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.181499 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.184004 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.184430 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.184524 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.188230 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:33.188239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:33.188247 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:33.188291 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:33.191932 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:33.192715 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:33.192724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:33.192731 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:33.192888 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:33.193351 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:33.193359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:33.193365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:33.193744 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:33.193903 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:33.194558 (+ 655us) server_negotiation.cc:299] Negotiation successful
0923 20:26:33.194837 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":258,"threads_started":1}
I20260923 20:26:33.195902 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.175010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46347 (local address 127.14.127.194:43331)
0923 20:26:33.175859 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:33.175907 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:33.178233 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:33.179007 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:33.179021 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:33.179821 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:33.180594 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.180626 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.181686 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.181694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.183865 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.183882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.184651 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.184660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.184978 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.185705 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:33.185741 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:33.187983 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:33.192063 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:33.192090 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:33.192108 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:33.192579 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:33.192994 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:33.192997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:33.193000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:33.193209 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:33.193863 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:33.193869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:33.194170 (+ 301us) client_negotiation.cc:769] Sending connection context
0923 20:26:33.194386 (+ 216us) client_negotiation.cc:240] Negotiation successful
0923 20:26:33.194817 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":389,"threads_started":1}
I20260923 20:26:33.197466 945 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46347
I20260923 20:26:33.197798 945 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:33.198792 945 heartbeater.cc:511] Master 127.14.127.254:46347 requested a full tablet report, sending...
I20260923 20:26:33.200477 622 ts_manager.cc:194] Registered new tserver with Master: c5f09bd1b73a4242807c8f87b33ad7bf (127.14.127.194:39021)
I20260923 20:26:33.201293 622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:43331
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:26:33.233554 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46347
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:34255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:33.558266 951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:33.558599 951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:33.558802 951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:33.566184 951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:33.566334 951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:33.566512 951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:26:33.576408 951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:34255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:33.577749 951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:33.579164 951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:33.592423 956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:33.592500 957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:33.592521 959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:33.593478 951 server_base.cc:1034] running on GCE node
I20260923 20:26:33.594051 951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:33.595033 951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:33.596271 951 hybrid_clock.cc:648] HybridClock initialized: now 1790195193596211 us; error 60 us; skew 500 ppm
I20260923 20:26:33.596567 951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:33 dist-test-slave-wmd0 krb5kdc[577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195193, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:33.600003 951 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:26:33.601459 951 webserver.cc:460] Webserver started at http://127.14.127.195:44483/ using document root <none> and password file <none>
I20260923 20:26:33.602195 951 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:33.602308 951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:33.602583 951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:33.605847 951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "7aee81df13044d37aaac5f4a3c0a4916"
format_stamp: "Formatted at 2026-09-23 20:26:33 on dist-test-slave-wmd0"
server_key: "3b7542e04d31ad6a4df7d2d6b99ab0b7"
server_key_iv: "e96f104993fe4ecac9610103f648c72e"
server_key_version: "encryptionkey@0"
I20260923 20:26:33.606757 951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "7aee81df13044d37aaac5f4a3c0a4916"
format_stamp: "Formatted at 2026-09-23 20:26:33 on dist-test-slave-wmd0"
server_key: "3b7542e04d31ad6a4df7d2d6b99ab0b7"
server_key_iv: "e96f104993fe4ecac9610103f648c72e"
server_key_version: "encryptionkey@0"
I20260923 20:26:33.612112 951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:26:33.615696 966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:33.617030 951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:33.617228 951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "7aee81df13044d37aaac5f4a3c0a4916"
format_stamp: "Formatted at 2026-09-23 20:26:33 on dist-test-slave-wmd0"
server_key: "3b7542e04d31ad6a4df7d2d6b99ab0b7"
server_key_iv: "e96f104993fe4ecac9610103f648c72e"
server_key_version: "encryptionkey@0"
I20260923 20:26:33.617424 951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:33.657787 951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:33.661896 951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:33.662191 951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:33.663220 951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:33.664505 951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:33.664621 951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:33.664764 951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:33.664839 951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:33.682523 811 heartbeater.cc:503] Master 127.14.127.254:46347 was elected leader, sending a full tablet report...
I20260923 20:26:33.698838 951 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:45515
I20260923 20:26:33.698884 1078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:45515 every 8 connection(s)
I20260923 20:26:33.700325 951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:26:33.707264 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 951
I20260923 20:26:33.707456 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:26:33.708019 14847 external_mini_cluster.cc:1428] Setting key 115f68ca671b874067ddf8fc93b09a9d
Sep 23 20:26:33 dist-test-slave-wmd0 krb5kdc[577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195193, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:33.721365 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.702784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:47097 (local address 127.14.127.254:46347)
0923 20:26:33.703269 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:33.703274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:33.704228 (+ 954us) server_negotiation.cc:407] Connection header received
0923 20:26:33.706140 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:33.706146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:33.706241 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:33.706362 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:33.708466 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.709414 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.711346 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.711721 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.711820 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.715138 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:33.715142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:33.715146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:33.715176 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:33.718238 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:33.719073 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:33.719078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:33.719082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:33.719206 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:33.719703 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:33.719709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:33.719713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:33.720082 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:33.720215 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:33.720886 (+ 671us) server_negotiation.cc:299] Negotiation successful
0923 20:26:33.721046 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":218,"threads_started":1}
I20260923 20:26:33.722735 1082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.703371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46347 (local address 127.14.127.195:47097)
0923 20:26:33.704061 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:33.704099 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:33.705827 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:33.706602 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:33.706658 (+ 56us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:33.707446 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:33.708280 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.708308 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.709561 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.709566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.711218 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.711230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.711900 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.711904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.712086 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.712782 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:33.712804 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:33.714946 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:33.718405 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:33.718415 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:33.718431 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:33.718940 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:33.719319 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:33.719324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:33.719328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:33.719560 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:33.720225 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:33.720233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:33.720550 (+ 317us) client_negotiation.cc:769] Sending connection context
0923 20:26:33.720791 (+ 241us) client_negotiation.cc:240] Negotiation successful
0923 20:26:33.721279 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":266,"threads_started":1}
I20260923 20:26:33.724048 1079 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46347
I20260923 20:26:33.724313 1079 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:33.724962 1079 heartbeater.cc:511] Master 127.14.127.254:46347 requested a full tablet report, sending...
I20260923 20:26:33.726245 622 ts_manager.cc:194] Registered new tserver with Master: 7aee81df13044d37aaac5f4a3c0a4916 (127.14.127.195:45515)
I20260923 20:26:33.727100 622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:47097
I20260923 20:26:33.735460 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:26:33.751497 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.739465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.14.127.254:46347)
0923 20:26:33.739716 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:33.739725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:33.740049 (+ 324us) server_negotiation.cc:407] Connection header received
0923 20:26:33.740176 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:33.740180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:33.740236 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:33.740314 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:33.741594 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.742287 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.743575 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.743965 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.744056 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.746534 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:33.746538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:33.746542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:33.746569 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:33.748917 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:33.749620 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:33.749623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:33.749626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:33.749742 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:33.750131 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:33.750134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:33.750136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:33.750429 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:33.750695 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:33.751016 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 20:26:33.751181 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:26:33.755455 622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:26:33.762436 622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:26:33.794872 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.774496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:39021 (local address 127.0.0.1:58468)
0923 20:26:33.784797 (+ 10301us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:33.784848 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:33.785050 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:33.786557 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:33.786562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:33.786590 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:33.787263 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.787283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.789099 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.789105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.791651 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.791665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.793803 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.793810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.793994 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.794044 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 20:26:33.794178 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:26:33.794364 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9942,"mutex_wait_us":51,"thread_start_us":11572,"threads_started":1}
I20260923 20:26:33.795277 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.775017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:45515 (local address 127.0.0.1:58658)
0923 20:26:33.786257 (+ 11240us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:33.786275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:33.786389 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:33.788188 (+ 1799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:33.788193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:33.788219 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:33.788644 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.788655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.790695 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.790700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.792616 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.792628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.794413 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.794419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.794594 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.794685 (+ 91us) client_negotiation.cc:769] Sending connection context
0923 20:26:33.794780 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 20:26:33.794913 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11114,"mutex_wait_us":4,"thread_start_us":11839,"threads_started":1}
I20260923 20:26:33.795432 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.774763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.14.127.194:39021)
0923 20:26:33.785198 (+ 10435us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:33.785209 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:33.785242 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:26:33.785356 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:33.785366 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:33.785654 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:33.785860 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:33.787498 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.788938 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.791817 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.793315 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.793485 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.793676 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:33.794729 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0923 20:26:33.794906 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10239,"thread_start_us":10928,"threads_started":1}
I20260923 20:26:33.796387 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.775131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.14.127.195:45515)
0923 20:26:33.787735 (+ 12604us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:33.787741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:33.787767 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:26:33.787837 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:33.787842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:33.788006 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:33.788152 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:33.788852 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.790515 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.792775 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.794203 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.794415 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.794575 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:33.795892 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0923 20:26:33.796052 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12473,"thread_start_us":12309,"threads_started":1}
I20260923 20:26:33.796449 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.776629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44783 (local address 127.0.0.1:46056)
0923 20:26:33.787171 (+ 10542us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:33.787187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:33.788368 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:33.788943 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:33.788947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:33.788972 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:33.789475 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.789486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.791217 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.791224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.793474 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.793489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.795677 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.795682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.795829 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.795850 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:26:33.795956 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:26:33.796075 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10399,"mutex_wait_us":1,"spinlock_wait_cycles":5376,"thread_start_us":7625,"threads_started":1}
I20260923 20:26:33.796496 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.777346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.14.127.193:44783)
0923 20:26:33.785684 (+ 8338us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:33.785696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:33.787285 (+ 1589us) server_negotiation.cc:407] Connection header received
0923 20:26:33.788561 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:33.788568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:33.788733 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:33.788944 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:33.789677 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.791064 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.793665 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.795148 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.795302 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.795476 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:33.796000 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 20:26:33.796182 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8154,"thread_start_us":8017,"threads_started":1}
I20260923 20:26:33.800093 746 tablet_service.cc:1467] Processing CreateTablet for tablet caffb62b9ca04a0794a63e3ded1dbf19 (DEFAULT_TABLE table=test-table [id=8defa734374d4b81848bf20bfee9d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:33.800880 1014 tablet_service.cc:1467] Processing CreateTablet for tablet caffb62b9ca04a0794a63e3ded1dbf19 (DEFAULT_TABLE table=test-table [id=8defa734374d4b81848bf20bfee9d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:33.801403 746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caffb62b9ca04a0794a63e3ded1dbf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:33.802058 1014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caffb62b9ca04a0794a63e3ded1dbf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:33.807085 880 tablet_service.cc:1467] Processing CreateTablet for tablet caffb62b9ca04a0794a63e3ded1dbf19 (DEFAULT_TABLE table=test-table [id=8defa734374d4b81848bf20bfee9d526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:33.808418 880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caffb62b9ca04a0794a63e3ded1dbf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:33.812924 1098 tablet_bootstrap.cc:492] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: Bootstrap starting.
I20260923 20:26:33.814901 1099 tablet_bootstrap.cc:492] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Bootstrap starting.
I20260923 20:26:33.816644 1098 tablet_bootstrap.cc:654] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:33.817986 1098 log.cc:824] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:33.818224 1099 tablet_bootstrap.cc:654] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:33.819272 1099 log.cc:824] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:33.821072 1098 tablet_bootstrap.cc:492] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: No bootstrap required, opened a new log
I20260923 20:26:33.821339 1098 ts_tablet_manager.cc:1397] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260923 20:26:33.822237 1099 tablet_bootstrap.cc:492] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: No bootstrap required, opened a new log
I20260923 20:26:33.822266 1102 tablet_bootstrap.cc:492] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Bootstrap starting.
I20260923 20:26:33.822903 1099 ts_tablet_manager.cc:1397] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260923 20:26:33.825278 1102 tablet_bootstrap.cc:654] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:33.826319 1102 log.cc:824] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:33.828739 1102 tablet_bootstrap.cc:492] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: No bootstrap required, opened a new log
I20260923 20:26:33.829001 1102 ts_tablet_manager.cc:1397] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260923 20:26:33.832047 1098 raft_consensus.cc:348] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.832473 1098 raft_consensus.cc:374] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:33.832566 1098 raft_consensus.cc:729] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ad8565bd144f25b234dc6101d0b727, State: Initialized, Role: FOLLOWER
I20260923 20:26:33.833178 1098 consensus_queue.cc:260] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.833180 1099 raft_consensus.cc:348] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.833554 1099 raft_consensus.cc:374] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:33.833671 1099 raft_consensus.cc:729] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aee81df13044d37aaac5f4a3c0a4916, State: Initialized, Role: FOLLOWER
I20260923 20:26:33.834340 1099 consensus_queue.cc:260] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.835088 1098 ts_tablet_manager.cc:1428] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260923 20:26:33.836020 1079 heartbeater.cc:503] Master 127.14.127.254:46347 was elected leader, sending a full tablet report...
I20260923 20:26:33.836779 1099 ts_tablet_manager.cc:1428] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260923 20:26:33.837136 1102 raft_consensus.cc:348] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.837523 1102 raft_consensus.cc:374] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:33.837625 1102 raft_consensus.cc:729] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f09bd1b73a4242807c8f87b33ad7bf, State: Initialized, Role: FOLLOWER
I20260923 20:26:33.838171 1102 consensus_queue.cc:260] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.839268 945 heartbeater.cc:503] Master 127.14.127.254:46347 was elected leader, sending a full tablet report...
I20260923 20:26:33.839838 1102 ts_tablet_manager.cc:1428] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
W20260923 20:26:33.903208 812 tablet.cc:2369] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:26:33.923724 946 tablet.cc:2369] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:26:33.951993 1080 tablet.cc:2369] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:26:33.966040 1104 raft_consensus.cc:482] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:26:33.966374 1104 raft_consensus.cc:504] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.968153 1104 leader_election.cc:288] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5f09bd1b73a4242807c8f87b33ad7bf (127.14.127.194:39021), 7aee81df13044d37aaac5f4a3c0a4916 (127.14.127.195:45515)
I20260923 20:26:33.975066 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.968625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:49607 (local address 127.14.127.195:45515)
0923 20:26:33.968840 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:33.968848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:33.969629 (+ 781us) server_negotiation.cc:407] Connection header received
0923 20:26:33.969802 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:33.969805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:33.969884 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:33.969976 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:33.970642 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.971355 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.973172 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.974023 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.974151 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.974205 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:33.974669 (+ 464us) server_negotiation.cc:299] Negotiation successful
0923 20:26:33.974815 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:26:33.975224 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.968546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:60477 (local address 127.14.127.194:39021)
0923 20:26:33.968832 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:33.968841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:33.969529 (+ 688us) server_negotiation.cc:407] Connection header received
0923 20:26:33.969728 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:33.969736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:33.969813 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:33.969890 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:33.970701 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.971672 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.973388 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.974237 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.974390 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.974461 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:33.974827 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 20:26:33.974967 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260923 20:26:33.975359 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.968628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:45515 (local address 127.14.127.193:49607)
0923 20:26:33.969470 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:33.969513 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:33.969668 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:33.970012 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:33.970019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:33.970045 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:33.970452 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.970465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.971488 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.971494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.973018 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.973029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.974238 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.974245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.974421 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.974484 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 20:26:33.974569 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 20:26:33.974733 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":22,"thread_start_us":418,"threads_started":1}
I20260923 20:26:33.975512 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:33.968422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:39021 (local address 127.14.127.193:60477)
0923 20:26:33.969408 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:33.969425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:33.969552 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:33.969938 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:33.969945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:33.969973 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:33.970515 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.970526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.971802 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.971806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.973243 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:33.973256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:33.974431 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:33.974436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:33.974651 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:33.974686 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:26:33.974770 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 20:26:33.974889 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":58,"thread_start_us":685,"threads_started":1}
I20260923 20:26:33.976557 1034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caffb62b9ca04a0794a63e3ded1dbf19" candidate_uuid: "85ad8565bd144f25b234dc6101d0b727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aee81df13044d37aaac5f4a3c0a4916" is_pre_election: true
I20260923 20:26:33.976555 900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caffb62b9ca04a0794a63e3ded1dbf19" candidate_uuid: "85ad8565bd144f25b234dc6101d0b727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" is_pre_election: true
I20260923 20:26:33.977110 900 raft_consensus.cc:2454] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ad8565bd144f25b234dc6101d0b727 in term 0.
I20260923 20:26:33.977104 1034 raft_consensus.cc:2454] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ad8565bd144f25b234dc6101d0b727 in term 0.
I20260923 20:26:33.977778 700 leader_election.cc:302] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aee81df13044d37aaac5f4a3c0a4916, 85ad8565bd144f25b234dc6101d0b727; no voters:
I20260923 20:26:33.978199 1104 raft_consensus.cc:2781] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:26:33.978346 1104 raft_consensus.cc:482] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:26:33.978466 1104 raft_consensus.cc:3037] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:33.980677 1104 raft_consensus.cc:504] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.981321 1104 leader_election.cc:288] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [CANDIDATE]: Term 1 election: Requested vote from peers c5f09bd1b73a4242807c8f87b33ad7bf (127.14.127.194:39021), 7aee81df13044d37aaac5f4a3c0a4916 (127.14.127.195:45515)
I20260923 20:26:33.981648 900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caffb62b9ca04a0794a63e3ded1dbf19" candidate_uuid: "85ad8565bd144f25b234dc6101d0b727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf"
I20260923 20:26:33.981838 900 raft_consensus.cc:3037] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:33.981839 1034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caffb62b9ca04a0794a63e3ded1dbf19" candidate_uuid: "85ad8565bd144f25b234dc6101d0b727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aee81df13044d37aaac5f4a3c0a4916"
I20260923 20:26:33.982036 1034 raft_consensus.cc:3037] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:33.984031 900 raft_consensus.cc:2454] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ad8565bd144f25b234dc6101d0b727 in term 1.
I20260923 20:26:33.984251 1034 raft_consensus.cc:2454] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ad8565bd144f25b234dc6101d0b727 in term 1.
I20260923 20:26:33.984431 699 leader_election.cc:302] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ad8565bd144f25b234dc6101d0b727, c5f09bd1b73a4242807c8f87b33ad7bf; no voters:
I20260923 20:26:33.984722 1104 raft_consensus.cc:2781] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:33.985100 1104 raft_consensus.cc:686] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 1 LEADER]: Becoming Leader. State: Replica: 85ad8565bd144f25b234dc6101d0b727, State: Running, Role: LEADER
I20260923 20:26:33.985561 1104 consensus_queue.cc:237] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } }
I20260923 20:26:33.990171 620 catalog_manager.cc:5495] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85ad8565bd144f25b234dc6101d0b727 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "85ad8565bd144f25b234dc6101d0b727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ad8565bd144f25b234dc6101d0b727" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 44783 } health_report { overall_health: HEALTHY } } }
I20260923 20:26:34.034394 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.029554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.14.127.193:44783)
0923 20:26:34.029733 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:34.029737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:34.029753 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:26:34.029832 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:34.029835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:34.029920 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:34.030063 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:26:34.030420 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:34.031165 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:34.032522 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:34.032898 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:34.033000 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:34.033434 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:26:34.033904 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:26:34.034030 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:34.034162 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 20:26:34.034200 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:26:34.042692 1034 raft_consensus.cc:1261] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 1 FOLLOWER]: Refusing update from remote peer 85ad8565bd144f25b234dc6101d0b727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:34.042768 900 raft_consensus.cc:1261] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 1 FOLLOWER]: Refusing update from remote peer 85ad8565bd144f25b234dc6101d0b727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:34.043751 1109 consensus_queue.cc:1035] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aee81df13044d37aaac5f4a3c0a4916" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:34.044157 1104 consensus_queue.cc:1035] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f09bd1b73a4242807c8f87b33ad7bf" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:34.083117 620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:34.083504 620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:34.088146 620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a72f178750e43929db4ce45c8da5f9f: Sending DeleteTablet for 3 replicas of tablet caffb62b9ca04a0794a63e3ded1dbf19
I20260923 20:26:34.089293 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet caffb62b9ca04a0794a63e3ded1dbf19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:34 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:58658
I20260923 20:26:34.090513 880 tablet_service.cc:1514] Processing DeleteTablet for tablet caffb62b9ca04a0794a63e3ded1dbf19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:34 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:58468
I20260923 20:26:34.091598 746 tablet_service.cc:1514] Processing DeleteTablet for tablet caffb62b9ca04a0794a63e3ded1dbf19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:34 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:46056
I20260923 20:26:34.096966 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 683
I20260923 20:26:34.102449 1123 tablet_replica.cc:330] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: stopping tablet replica
I20260923 20:26:34.103003 1123 raft_consensus.cc:2229] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:34.103470 1123 raft_consensus.cc:2258] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 20:26:34 dist-test-slave-wmd0 krb5kdc[577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:26:34.120394 1123 ts_tablet_manager.cc:1905] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:34.121671 1122 tablet_replica.cc:330] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: stopping tablet replica
I20260923 20:26:34.124315 1124 tablet_replica.cc:330] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: stopping tablet replica
I20260923 20:26:34.124718 1124 raft_consensus.cc:2229] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:26:34.125377 1124 raft_consensus.cc:2258] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:34.127871 1123 ts_tablet_manager.cc:1918] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:34.128108 1123 log.cc:1196] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/caffb62b9ca04a0794a63e3ded1dbf19
I20260923 20:26:34.128396 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.098299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.14.127.193:44783)
0923 20:26:34.098473 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:34.098481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:34.099223 (+ 742us) server_negotiation.cc:407] Connection header received
0923 20:26:34.099427 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:34.099432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:34.099507 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:34.099591 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:34.105580 (+ 5989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:34.107009 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:34.109367 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:34.110249 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:34.110357 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:34.118568 (+ 8211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:34.118585 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:34.118600 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:34.118684 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:34.123170 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:34.125761 (+ 2591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:34.125773 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:34.125778 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:34.125942 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:34.126355 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:34.126361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:34.126363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:34.126864 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:34.127174 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:34.127728 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 20:26:34.127974 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:26:34.128129 1124 ts_tablet_manager.cc:1905] T caffb62b9ca04a0794a63e3ded1dbf19 P 85ad8565bd144f25b234dc6101d0b727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:34.128999 1123 ts_tablet_manager.cc:1939] T caffb62b9ca04a0794a63e3ded1dbf19 P c5f09bd1b73a4242807c8f87b33ad7bf: Deleting consensus metadata
I20260923 20:26:34.129423 806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:34.129758 1122 raft_consensus.cc:2229] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:34.131464 1122 raft_consensus.cc:2258] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:34.131705 606 catalog_manager.cc:4841] TS c5f09bd1b73a4242807c8f87b33ad7bf (127.14.127.194:39021): tablet caffb62b9ca04a0794a63e3ded1dbf19 (table test-table [id=8defa734374d4b81848bf20bfee9d526]) successfully deleted
I20260923 20:26:34.134382 1122 ts_tablet_manager.cc:1905] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:34.141847 1122 ts_tablet_manager.cc:1918] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:34.142025 1122 log.cc:1196] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/caffb62b9ca04a0794a63e3ded1dbf19
I20260923 20:26:34.142540 1122 ts_tablet_manager.cc:1939] T caffb62b9ca04a0794a63e3ded1dbf19 P 7aee81df13044d37aaac5f4a3c0a4916: Deleting consensus metadata
I20260923 20:26:34.144440 607 catalog_manager.cc:4841] TS 7aee81df13044d37aaac5f4a3c0a4916 (127.14.127.195:45515): tablet caffb62b9ca04a0794a63e3ded1dbf19 (table test-table [id=8defa734374d4b81848bf20bfee9d526]) successfully deleted
I20260923 20:26:34.229228 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 683
W20260923 20:26:34.247318 609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.127.193:44783 (error 108)
I20260923 20:26:34.247394 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 817
I20260923 20:26:34.249169 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.248739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44783 (local address 127.0.0.1:46086)
0923 20:26:34.248937 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:34.249030 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260923 20:26:34.249375 609 catalog_manager.cc:4584] TS 85ad8565bd144f25b234dc6101d0b727 (127.14.127.193:44783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet caffb62b9ca04a0794a63e3ded1dbf19: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44783: connect: Connection refused (error 111)
Sep 23 20:26:34 dist-test-slave-wmd0 krb5kdc[577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:26:34.261688 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.248284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.14.127.194:39021)
0923 20:26:34.248478 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:34.248484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:34.248505 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:26:34.248582 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:34.248586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:34.248663 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:34.248747 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:34.250181 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:34.250974 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:34.252192 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:34.252554 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:34.252654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:34.256684 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:34.256696 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:34.256709 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:34.256743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:34.259419 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:34.260043 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:34.260050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:34.260052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:34.260166 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:34.260476 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:34.260479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:34.260481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:34.260837 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:34.261036 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:34.261285 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 20:26:34.261433 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:26:34.262338 940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:34.302753 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.302274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44783 (local address 127.0.0.1:46092)
0923 20:26:34.302468 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:34.302560 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:26:34.303725 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.303391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44783 (local address 127.0.0.1:46104)
0923 20:26:34.303536 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:34.303621 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 20:26:34.351210 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 817
I20260923 20:26:34.364435 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.363808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44783 (local address 127.0.0.1:46118)
0923 20:26:34.364073 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:34.364209 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 20:26:34.365563 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.365133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44783 (local address 127.0.0.1:46132)
0923 20:26:34.365313 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:34.365405 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 20:26:34.369257 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 951
Sep 23 20:26:34 dist-test-slave-wmd0 krb5kdc[577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:26:34.383886 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.369979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.14.127.195:45515)
0923 20:26:34.370161 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:34.370167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:34.370186 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:26:34.370351 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:34.370354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:34.370425 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:34.370527 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:34.371878 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:34.372615 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:34.373926 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:34.374328 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:34.374464 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:34.378882 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:34.378897 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:34.378913 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:34.378946 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:34.381447 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:34.382051 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:34.382057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:34.382059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:34.382187 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:34.382503 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:34.382506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:34.382508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:34.383040 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:34.383257 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:34.383489 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 20:26:34.383644 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:26:34.384474 1074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:34.446466 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.445837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44783 (local address 127.0.0.1:46142)
0923 20:26:34.446170 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:34.446275 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260923 20:26:34.447549 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:34.447202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:44783 (local address 127.0.0.1:46148)
0923 20:26:34.447379 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:34.447446 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:44783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260923 20:26:34.473929 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 951
I20260923 20:26:34.490545 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 590
I20260923 20:26:34.491840 652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:34.602710 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 590
2026-09-23T20:26:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4354 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T20:26:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:34Z Disabled control of system clock
I20260923 20:26:34.680958 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46391
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:46391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:26:35.012032 1134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:35.012320 1134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:35.012486 1134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:35.019829 1134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:35.019932 1134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:35.019976 1134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:35.020059 1134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:35.020123 1134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:35.029296 1134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:46391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:35.030645 1134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:35.032420 1134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:35.044489 1142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:35.044571 1140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:35.044548 1139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:35.045472 1134 server_base.cc:1034] running on GCE node
I20260923 20:26:35.046249 1134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:35.047696 1134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:35.048931 1134 hybrid_clock.cc:648] HybridClock initialized: now 1790195195048896 us; error 48 us; skew 500 ppm
I20260923 20:26:35.049290 1134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:35.056685 1134 webserver.cc:460] Webserver started at http://127.14.127.254:33315/ using document root <none> and password file <none>
I20260923 20:26:35.057405 1134 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:35.057541 1134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:35.057870 1134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:35.060879 1134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "20538364e14e4a9db6be14df6e34e98f"
format_stamp: "Formatted at 2026-09-23 20:26:35 on dist-test-slave-wmd0"
server_key: "06044f8349c28b9dda7bcf4ed9a852fe"
server_key_iv: "6cc2b24ed68e47eee8ad67ba0767b5c1"
server_key_version: "encryptionkey@0"
I20260923 20:26:35.061704 1134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "20538364e14e4a9db6be14df6e34e98f"
format_stamp: "Formatted at 2026-09-23 20:26:35 on dist-test-slave-wmd0"
server_key: "06044f8349c28b9dda7bcf4ed9a852fe"
server_key_iv: "6cc2b24ed68e47eee8ad67ba0767b5c1"
server_key_version: "encryptionkey@0"
I20260923 20:26:35.066994 1134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 20:26:35.070483 1148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:35.071897 1134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:26:35.072084 1134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "20538364e14e4a9db6be14df6e34e98f"
format_stamp: "Formatted at 2026-09-23 20:26:35 on dist-test-slave-wmd0"
server_key: "06044f8349c28b9dda7bcf4ed9a852fe"
server_key_iv: "6cc2b24ed68e47eee8ad67ba0767b5c1"
server_key_version: "encryptionkey@0"
I20260923 20:26:35.072311 1134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:35.091274 1134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:35.092118 1134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:35.092414 1134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:35.111155 1134 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:46391
I20260923 20:26:35.111183 1199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:46391 every 8 connection(s)
I20260923 20:26:35.112519 1134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:35.115024 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 1134
I20260923 20:26:35.115224 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:35.115698 14847 external_mini_cluster.cc:1428] Setting key 2c2e65a963e8a1b7f051e564f38278d4
I20260923 20:26:35.116492 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:35.131358 1200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f: Bootstrap starting.
I20260923 20:26:35.131506 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:35.117650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.14.127.254:46391)
0923 20:26:35.118734 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:35.118750 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:35.118800 (+ 50us) server_negotiation.cc:407] Connection header received
0923 20:26:35.121175 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:35.121203 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:35.121808 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:35.122260 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:35.122825 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:35.124807 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:35.126283 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:35.127399 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:35.127606 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:35.128722 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:35.128734 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:35.128758 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:35.129496 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:35.129588 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:35.129613 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:35.129725 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:35.129910 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 20:26:35.130237 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":363,"threads_started":1}
I20260923 20:26:35.134562 1200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:35.135646 1200 log.cc:824] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:35.138393 1200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f: No bootstrap required, opened a new log
I20260923 20:26:35.145671 1200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20538364e14e4a9db6be14df6e34e98f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46391 } }
I20260923 20:26:35.146037 1200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:35.146163 1200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20538364e14e4a9db6be14df6e34e98f, State: Initialized, Role: FOLLOWER
I20260923 20:26:35.146773 1200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20538364e14e4a9db6be14df6e34e98f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46391 } }
I20260923 20:26:35.147024 1200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:35.147186 1200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:35.147347 1200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:35.149712 1200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20538364e14e4a9db6be14df6e34e98f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46391 } }
I20260923 20:26:35.150197 1200 leader_election.cc:302] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20538364e14e4a9db6be14df6e34e98f; no voters:
I20260923 20:26:35.150833 1200 leader_election.cc:288] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:35.150985 1205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:35.151727 1205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [term 1 LEADER]: Becoming Leader. State: Replica: 20538364e14e4a9db6be14df6e34e98f, State: Running, Role: LEADER
I20260923 20:26:35.152279 1205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20538364e14e4a9db6be14df6e34e98f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46391 } }
I20260923 20:26:35.152819 1200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:35.155094 1207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20538364e14e4a9db6be14df6e34e98f. Latest consensus state: current_term: 1 leader_uuid: "20538364e14e4a9db6be14df6e34e98f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20538364e14e4a9db6be14df6e34e98f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46391 } } }
I20260923 20:26:35.155184 1206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20538364e14e4a9db6be14df6e34e98f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20538364e14e4a9db6be14df6e34e98f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46391 } } }
I20260923 20:26:35.155483 1207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:35.155556 1206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:35.156940 1211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:35.162504 1211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:35.171480 1211 catalog_manager.cc:1347] Generated new cluster ID: 2d9d0fedfeef441b91963b93e6fe9abd
I20260923 20:26:35.171595 1211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:35.181802 1211 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:35.183252 1211 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:35.197310 1211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f: Generated new TSK 0
I20260923 20:26:35.198179 1211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:35.214586 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46391
--builtin_ntp_servers=127.14.127.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:26:35.551319 1224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:35.551599 1224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:35.551692 1224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:35.559423 1224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:35.559526 1224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:35.559630 1224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:26:35.568799 1224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:35.570046 1224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:35.571863 1224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:35.584914 1229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:35.584831 1232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:35.584928 1230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:35.585680 1224 server_base.cc:1034] running on GCE node
I20260923 20:26:35.586310 1224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:35.587314 1224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:35.588547 1224 hybrid_clock.cc:648] HybridClock initialized: now 1790195195588484 us; error 46 us; skew 500 ppm
I20260923 20:26:35.588917 1224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:35.591300 1224 webserver.cc:460] Webserver started at http://127.14.127.193:35029/ using document root <none> and password file <none>
I20260923 20:26:35.591991 1224 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:35.592120 1224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:35.592446 1224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:35.595371 1224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "0af092cbaaca4923bdd29df636462fd2"
format_stamp: "Formatted at 2026-09-23 20:26:35 on dist-test-slave-wmd0"
server_key: "0f2d15a18ef221b5acc23ddc1f82c81b"
server_key_iv: "c99d2358a7bdf77cb94359d1f3bce9d2"
server_key_version: "encryptionkey@0"
I20260923 20:26:35.596179 1224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "0af092cbaaca4923bdd29df636462fd2"
format_stamp: "Formatted at 2026-09-23 20:26:35 on dist-test-slave-wmd0"
server_key: "0f2d15a18ef221b5acc23ddc1f82c81b"
server_key_iv: "c99d2358a7bdf77cb94359d1f3bce9d2"
server_key_version: "encryptionkey@0"
I20260923 20:26:35.601208 1224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:26:35.604647 1238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:35.606012 1224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 20:26:35.606205 1224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "0af092cbaaca4923bdd29df636462fd2"
format_stamp: "Formatted at 2026-09-23 20:26:35 on dist-test-slave-wmd0"
server_key: "0f2d15a18ef221b5acc23ddc1f82c81b"
server_key_iv: "c99d2358a7bdf77cb94359d1f3bce9d2"
server_key_version: "encryptionkey@0"
I20260923 20:26:35.606429 1224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:35.620945 1224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:35.621744 1224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:35.622051 1224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:35.623023 1224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:35.624418 1224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:35.624549 1224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:35.624692 1224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:35.624784 1224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:35.657953 1224 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:38629
I20260923 20:26:35.658046 1350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:38629 every 8 connection(s)
I20260923 20:26:35.659360 1224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:26:35.669153 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 1224
I20260923 20:26:35.669337 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:26:35.670054 14847 external_mini_cluster.cc:1428] Setting key 25073f8ba4d80b9f86e817f635a8e231
I20260923 20:26:35.672996 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:35.661929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:48465 (local address 127.14.127.254:46391)
0923 20:26:35.662371 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:35.662377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:35.663626 (+ 1249us) server_negotiation.cc:407] Connection header received
0923 20:26:35.665406 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:35.665412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:35.665531 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:35.665656 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:35.667013 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:35.667807 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:35.669409 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:35.670128 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:35.670260 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:35.672021 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:35.672027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:35.672032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:35.672053 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:35.672103 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:35.672119 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:35.672224 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:35.672483 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 20:26:35.672601 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":201,"threads_started":1}
I20260923 20:26:35.673348 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46391
--builtin_ntp_servers=127.14.127.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:26:35.674230 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:35.662538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46391 (local address 127.14.127.193:48465)
0923 20:26:35.663474 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:35.663510 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:35.665064 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:35.665954 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:35.665967 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:35.666096 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:35.666829 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:35.666848 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:35.667944 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:35.667947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:35.669257 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:35.669273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:35.670627 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:35.670635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:35.670856 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:35.671642 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:35.671644 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:35.671690 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:35.671812 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:35.672239 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:35.672252 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:35.672260 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:26:35.672456 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 20:26:35.672888 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":270,"threads_started":1}
I20260923 20:26:35.675911 1351 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46391
I20260923 20:26:35.676256 1351 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:35.677078 1351 heartbeater.cc:511] Master 127.14.127.254:46391 requested a full tablet report, sending...
I20260923 20:26:35.679132 1165 ts_manager.cc:194] Registered new tserver with Master: 0af092cbaaca4923bdd29df636462fd2 (127.14.127.193:38629)
I20260923 20:26:35.680723 1165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.193:48465
W20260923 20:26:36.001374 1355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:36.001652 1355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:36.001737 1355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:36.008948 1355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:36.009061 1355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:36.009238 1355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:26:36.020875 1355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:36.022476 1355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:36.024914 1355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:36.038923 1363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:36.039044 1361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:36.039072 1360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:36.039924 1355 server_base.cc:1034] running on GCE node
I20260923 20:26:36.040562 1355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:36.041523 1355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:36.042725 1355 hybrid_clock.cc:648] HybridClock initialized: now 1790195196042656 us; error 56 us; skew 500 ppm
I20260923 20:26:36.043054 1355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:36.045347 1355 webserver.cc:460] Webserver started at http://127.14.127.194:45225/ using document root <none> and password file <none>
I20260923 20:26:36.046054 1355 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:36.046185 1355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:36.046523 1355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:36.049391 1355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "994734068b204972bd8d4f53a6ef6985"
format_stamp: "Formatted at 2026-09-23 20:26:36 on dist-test-slave-wmd0"
server_key: "47935106bad4d66c41e884ee84cdeed5"
server_key_iv: "f787303195627d89f2131654e138f786"
server_key_version: "encryptionkey@0"
I20260923 20:26:36.050184 1355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "994734068b204972bd8d4f53a6ef6985"
format_stamp: "Formatted at 2026-09-23 20:26:36 on dist-test-slave-wmd0"
server_key: "47935106bad4d66c41e884ee84cdeed5"
server_key_iv: "f787303195627d89f2131654e138f786"
server_key_version: "encryptionkey@0"
I20260923 20:26:36.055497 1355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 20:26:36.059122 1369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:36.060459 1355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:36.060647 1355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "994734068b204972bd8d4f53a6ef6985"
format_stamp: "Formatted at 2026-09-23 20:26:36 on dist-test-slave-wmd0"
server_key: "47935106bad4d66c41e884ee84cdeed5"
server_key_iv: "f787303195627d89f2131654e138f786"
server_key_version: "encryptionkey@0"
I20260923 20:26:36.060865 1355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:36.084765 1355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:36.085615 1355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:36.085935 1355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:36.086934 1355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:36.088418 1355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:36.088544 1355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:36.088677 1355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:36.088768 1355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:36.127539 1355 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:35501
I20260923 20:26:36.127591 1481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:35501 every 8 connection(s)
I20260923 20:26:36.129127 1355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:26:36.134667 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 1355
I20260923 20:26:36.134853 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:26:36.135388 14847 external_mini_cluster.cc:1428] Setting key 6db97b2c90fefc466bc2aec4aee7c4ff
I20260923 20:26:36.138204 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46391
--builtin_ntp_servers=127.14.127.212:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:26:36.142707 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.131968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:34819 (local address 127.14.127.254:46391)
0923 20:26:36.132200 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.132206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.133781 (+ 1575us) server_negotiation.cc:407] Connection header received
0923 20:26:36.136250 (+ 2469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.136256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.136348 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.136468 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:36.137606 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.138355 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.139977 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.140365 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.140501 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.141878 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:36.141883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:36.141887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:36.141903 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:36.141934 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:36.141946 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:36.142045 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.142271 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.142371 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:26:36.143750 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.132407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46391 (local address 127.14.127.194:34819)
0923 20:26:36.133600 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:36.133656 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:36.135986 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:36.136689 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:36.136697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:36.136790 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:36.137426 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.137454 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.138531 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.138535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.139814 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.139827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.140615 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.140626 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.140837 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.141552 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:36.141554 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:36.141592 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:36.141682 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:36.142045 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:36.142059 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:36.142068 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:26:36.142262 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 20:26:36.142689 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":353,"threads_started":1}
I20260923 20:26:36.145227 1482 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46391
I20260923 20:26:36.145525 1482 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:36.146211 1482 heartbeater.cc:511] Master 127.14.127.254:46391 requested a full tablet report, sending...
I20260923 20:26:36.147722 1165 ts_manager.cc:194] Registered new tserver with Master: 994734068b204972bd8d4f53a6ef6985 (127.14.127.194:35501)
I20260923 20:26:36.148960 1165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.194:34819
W20260923 20:26:36.467175 1485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:36.467456 1485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:36.467543 1485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:36.474865 1485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:36.474992 1485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:36.475164 1485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:26:36.484170 1485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:36.485430 1485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:36.487269 1485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:36.500177 1493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:36.500250 1490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:36.500238 1491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:36.504104 1485 server_base.cc:1034] running on GCE node
I20260923 20:26:36.504669 1485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:36.505545 1485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:36.506713 1485 hybrid_clock.cc:648] HybridClock initialized: now 1790195196506660 us; error 43 us; skew 500 ppm
I20260923 20:26:36.507047 1485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:36.509433 1485 webserver.cc:460] Webserver started at http://127.14.127.195:38849/ using document root <none> and password file <none>
I20260923 20:26:36.510149 1485 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:36.510267 1485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:36.510710 1485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:36.513653 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "63371bd0502d4026b100e925ecd73725"
format_stamp: "Formatted at 2026-09-23 20:26:36 on dist-test-slave-wmd0"
server_key: "477e1a6977dde64dd438932bff02c5da"
server_key_iv: "d292ab212a9e2723d9aa74106500f0be"
server_key_version: "encryptionkey@0"
I20260923 20:26:36.514431 1485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "63371bd0502d4026b100e925ecd73725"
format_stamp: "Formatted at 2026-09-23 20:26:36 on dist-test-slave-wmd0"
server_key: "477e1a6977dde64dd438932bff02c5da"
server_key_iv: "d292ab212a9e2723d9aa74106500f0be"
server_key_version: "encryptionkey@0"
I20260923 20:26:36.519636 1485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260923 20:26:36.523176 1499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:36.524468 1485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:26:36.524658 1485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "63371bd0502d4026b100e925ecd73725"
format_stamp: "Formatted at 2026-09-23 20:26:36 on dist-test-slave-wmd0"
server_key: "477e1a6977dde64dd438932bff02c5da"
server_key_iv: "d292ab212a9e2723d9aa74106500f0be"
server_key_version: "encryptionkey@0"
I20260923 20:26:36.524878 1485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:36.538048 1485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:36.538936 1485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:36.539237 1485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:36.540155 1485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:36.541548 1485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:36.541678 1485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:36.541806 1485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:36.541894 1485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:36.575549 1485 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:39763
I20260923 20:26:36.575631 1611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:39763 every 8 connection(s)
I20260923 20:26:36.577003 1485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:26:36.583938 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 1485
I20260923 20:26:36.584148 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:26:36.584719 14847 external_mini_cluster.cc:1428] Setting key 6d5430435df7cc67fe12b901d528eff0
I20260923 20:26:36.589579 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.579593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:43239 (local address 127.14.127.254:46391)
0923 20:26:36.579774 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.579779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.581252 (+ 1473us) server_negotiation.cc:407] Connection header received
0923 20:26:36.582948 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.582953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.583018 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.583105 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:36.584106 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.585212 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.587033 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.587576 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.587709 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.588928 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:36.588933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:36.588936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:36.588952 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:36.588980 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:36.588990 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:36.589056 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.589280 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.589353 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:26:36.590480 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.580122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46391 (local address 127.14.127.195:43239)
0923 20:26:36.581101 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:36.581142 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:36.582720 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:36.583292 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:36.583300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:36.583380 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:36.583922 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.583938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.585365 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.585370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.586888 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.586900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.587778 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.587783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.587971 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.588665 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:36.588667 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:36.588704 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:36.588775 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:36.589069 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:36.589087 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:36.589092 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:26:36.589226 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:26:36.589499 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":299,"threads_started":1}
I20260923 20:26:36.591823 1612 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46391
I20260923 20:26:36.592126 1612 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:36.592887 1612 heartbeater.cc:511] Master 127.14.127.254:46391 requested a full tablet report, sending...
I20260923 20:26:36.594203 1165 ts_manager.cc:194] Registered new tserver with Master: 63371bd0502d4026b100e925ecd73725 (127.14.127.195:39763)
I20260923 20:26:36.595078 1165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.195:43239
I20260923 20:26:36.599707 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:26:36.609233 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.603824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.14.127.254:46391)
0923 20:26:36.603980 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.603984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.604386 (+ 402us) server_negotiation.cc:407] Connection header received
0923 20:26:36.604510 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.604513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.604568 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.604651 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:36.605068 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.605734 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.607073 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.607489 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.607580 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.608650 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:36.608654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:36.608657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:36.608673 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:36.608709 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:36.608720 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:36.608824 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.608976 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.609017 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:26:36.613047 1165 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:26:36.619709 1165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:26:36.657207 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.632300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:38629 (local address 127.0.0.1:39148)
0923 20:26:36.647405 (+ 15105us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:36.647439 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:36.647604 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:36.649560 (+ 1956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:36.649568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:36.649598 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:36.650171 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.650185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.651943 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.651948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.654337 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.654351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.656266 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.656274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.656484 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.656542 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 20:26:36.656686 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 20:26:36.656879 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14890,"thread_start_us":17598,"threads_started":1}
I20260923 20:26:36.658515 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.631583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:39763 (local address 127.0.0.1:47358)
0923 20:26:36.633125 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:36.633166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:36.633316 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:36.651257 (+ 17941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:36.651263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:36.651293 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:36.651748 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.651759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.653930 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.653936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.655838 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.655848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.657636 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.657645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.657882 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.657917 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:26:36.658001 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 20:26:36.658144 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"mutex_wait_us":22,"spinlock_wait_cycles":1280,"thread_start_us":1181,"threads_started":1}
I20260923 20:26:36.658838 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.631875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.14.127.195:39763)
0923 20:26:36.650726 (+ 18851us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.650733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.650759 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:26:36.650832 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.650837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.651000 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.651946 (+ 946us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:36.652227 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.653767 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.656009 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.657392 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.657950 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.658109 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.658164 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.658278 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18662,"thread_start_us":18505,"threads_started":1}
I20260923 20:26:36.659400 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.633574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:35501 (local address 127.0.0.1:47512)
0923 20:26:36.650230 (+ 16656us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:36.650259 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:36.650368 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:36.651118 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:36.651125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:36.651153 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:36.651599 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.651614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.653039 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.653046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.655268 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.655283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.657458 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.657464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.658745 (+ 1281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.658788 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 20:26:36.658876 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 20:26:36.659019 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16511,"mutex_wait_us":171,"spinlock_wait_cycles":14464,"thread_start_us":14566,"threads_started":1}
I20260923 20:26:36.660082 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.633973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.14.127.194:35501)
0923 20:26:36.648113 (+ 14140us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.648123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.650489 (+ 2366us) server_negotiation.cc:407] Connection header received
0923 20:26:36.650634 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.650646 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.650903 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.651106 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:36.651809 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.652911 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.655476 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.656857 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.657004 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.657780 (+ 776us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.659613 (+ 1833us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.659786 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13982,"thread_start_us":13877,"threads_started":1}
I20260923 20:26:36.657308 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.632910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.14.127.193:38629)
0923 20:26:36.648835 (+ 15925us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.648844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.648876 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:26:36.648995 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.649004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.649326 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.649556 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:36.650416 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.651804 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.654596 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.656071 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.656241 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.656472 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.656762 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.656945 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15769,"thread_start_us":15635,"threads_started":1}
I20260923 20:26:36.664062 1417 tablet_service.cc:1467] Processing CreateTablet for tablet cc2bbdf53f7541059ccc27344a68cb9e (DEFAULT_TABLE table=test-table [id=abe9b0c2692540df9ebb0d72c3582a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:36.664062 1547 tablet_service.cc:1467] Processing CreateTablet for tablet cc2bbdf53f7541059ccc27344a68cb9e (DEFAULT_TABLE table=test-table [id=abe9b0c2692540df9ebb0d72c3582a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:36.665570 1417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2bbdf53f7541059ccc27344a68cb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:36.665570 1547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2bbdf53f7541059ccc27344a68cb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:36.665933 1286 tablet_service.cc:1467] Processing CreateTablet for tablet cc2bbdf53f7541059ccc27344a68cb9e (DEFAULT_TABLE table=test-table [id=abe9b0c2692540df9ebb0d72c3582a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:36.667184 1286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2bbdf53f7541059ccc27344a68cb9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:36.679363 1630 tablet_bootstrap.cc:492] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: Bootstrap starting.
I20260923 20:26:36.680783 1631 tablet_bootstrap.cc:492] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: Bootstrap starting.
I20260923 20:26:36.682818 1632 tablet_bootstrap.cc:492] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: Bootstrap starting.
I20260923 20:26:36.683754 1351 heartbeater.cc:503] Master 127.14.127.254:46391 was elected leader, sending a full tablet report...
I20260923 20:26:36.684312 1631 tablet_bootstrap.cc:654] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:36.684317 1630 tablet_bootstrap.cc:654] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:36.685621 1631 log.cc:824] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:36.685621 1630 log.cc:824] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:36.686120 1632 tablet_bootstrap.cc:654] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:36.688431 1632 log.cc:824] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:36.691303 1631 tablet_bootstrap.cc:492] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: No bootstrap required, opened a new log
I20260923 20:26:36.691578 1631 ts_tablet_manager.cc:1397] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260923 20:26:36.691608 1630 tablet_bootstrap.cc:492] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: No bootstrap required, opened a new log
I20260923 20:26:36.691773 1632 tablet_bootstrap.cc:492] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: No bootstrap required, opened a new log
I20260923 20:26:36.691874 1630 ts_tablet_manager.cc:1397] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260923 20:26:36.692011 1632 ts_tablet_manager.cc:1397] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:26:36.699563 1631 raft_consensus.cc:348] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.699934 1631 raft_consensus.cc:374] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:36.700028 1631 raft_consensus.cc:729] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994734068b204972bd8d4f53a6ef6985, State: Initialized, Role: FOLLOWER
I20260923 20:26:36.700690 1631 consensus_queue.cc:260] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.701790 1482 heartbeater.cc:503] Master 127.14.127.254:46391 was elected leader, sending a full tablet report...
I20260923 20:26:36.702512 1631 ts_tablet_manager.cc:1428] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260923 20:26:36.702553 1630 raft_consensus.cc:348] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.702546 1632 raft_consensus.cc:348] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.702967 1632 raft_consensus.cc:374] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:36.702968 1630 raft_consensus.cc:374] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:36.703064 1630 raft_consensus.cc:729] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af092cbaaca4923bdd29df636462fd2, State: Initialized, Role: FOLLOWER
I20260923 20:26:36.703060 1632 raft_consensus.cc:729] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63371bd0502d4026b100e925ecd73725, State: Initialized, Role: FOLLOWER
I20260923 20:26:36.703619 1632 consensus_queue.cc:260] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.703619 1630 consensus_queue.cc:260] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.705329 1630 ts_tablet_manager.cc:1428] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260923 20:26:36.705621 1632 ts_tablet_manager.cc:1428] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260923 20:26:36.705902 1612 heartbeater.cc:503] Master 127.14.127.254:46391 was elected leader, sending a full tablet report...
I20260923 20:26:36.710947 1636 raft_consensus.cc:482] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:26:36.711238 1636 raft_consensus.cc:504] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.712877 1636 leader_election.cc:288] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63371bd0502d4026b100e925ecd73725 (127.14.127.195:39763), 0af092cbaaca4923bdd29df636462fd2 (127.14.127.193:38629)
I20260923 20:26:36.719482 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.713390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:50821 (local address 127.14.127.195:39763)
0923 20:26:36.713547 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.713555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.714082 (+ 527us) server_negotiation.cc:407] Connection header received
0923 20:26:36.714315 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.714320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.714393 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.714488 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:36.715101 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.715800 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.717765 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.718524 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.718732 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.718800 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.719116 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.719244 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 20:26:36.719516 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.713234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:44565 (local address 127.14.127.193:38629)
0923 20:26:36.713457 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.713463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.714180 (+ 717us) server_negotiation.cc:407] Connection header received
0923 20:26:36.714336 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.714342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.714416 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.714505 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:36.715172 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.715836 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.717760 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.718514 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.718707 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.718793 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.719139 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.719263 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:26:36.719695 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.713023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:39763 (local address 127.14.127.194:50821)
0923 20:26:36.713985 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:36.714000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:36.714177 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:36.714505 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:36.714510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:36.714532 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:36.714968 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.714977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.715925 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.715932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.717568 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.717598 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.718766 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.718773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.718920 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.718979 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 20:26:36.719080 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:26:36.719210 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":81,"thread_start_us":640,"threads_started":1}
I20260923 20:26:36.719910 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.713299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:38629 (local address 127.14.127.194:44565)
0923 20:26:36.714076 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:36.714090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:36.714201 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:36.714533 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:36.714537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:36.714559 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:36.715004 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.715014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.715974 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.715984 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.717554 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:36.717565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.718786 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.718792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:36.718981 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.719003 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:26:36.719102 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 20:26:36.719243 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":96,"thread_start_us":404,"threads_started":1}
I20260923 20:26:36.720752 1567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2bbdf53f7541059ccc27344a68cb9e" candidate_uuid: "994734068b204972bd8d4f53a6ef6985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63371bd0502d4026b100e925ecd73725" is_pre_election: true
I20260923 20:26:36.720747 1306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2bbdf53f7541059ccc27344a68cb9e" candidate_uuid: "994734068b204972bd8d4f53a6ef6985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af092cbaaca4923bdd29df636462fd2" is_pre_election: true
I20260923 20:26:36.721212 1306 raft_consensus.cc:2454] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 994734068b204972bd8d4f53a6ef6985 in term 0.
I20260923 20:26:36.721235 1567 raft_consensus.cc:2454] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 994734068b204972bd8d4f53a6ef6985 in term 0.
I20260923 20:26:36.721827 1370 leader_election.cc:302] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af092cbaaca4923bdd29df636462fd2, 994734068b204972bd8d4f53a6ef6985; no voters:
I20260923 20:26:36.722209 1636 raft_consensus.cc:2781] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:26:36.722355 1636 raft_consensus.cc:482] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:26:36.722461 1636 raft_consensus.cc:3037] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:36.724598 1636 raft_consensus.cc:504] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.725135 1636 leader_election.cc:288] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [CANDIDATE]: Term 1 election: Requested vote from peers 63371bd0502d4026b100e925ecd73725 (127.14.127.195:39763), 0af092cbaaca4923bdd29df636462fd2 (127.14.127.193:38629)
I20260923 20:26:36.725557 1306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2bbdf53f7541059ccc27344a68cb9e" candidate_uuid: "994734068b204972bd8d4f53a6ef6985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af092cbaaca4923bdd29df636462fd2"
I20260923 20:26:36.725558 1567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2bbdf53f7541059ccc27344a68cb9e" candidate_uuid: "994734068b204972bd8d4f53a6ef6985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63371bd0502d4026b100e925ecd73725"
I20260923 20:26:36.725756 1306 raft_consensus.cc:3037] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:36.725756 1567 raft_consensus.cc:3037] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:36.729034 1306 raft_consensus.cc:2454] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994734068b204972bd8d4f53a6ef6985 in term 1.
I20260923 20:26:36.729034 1567 raft_consensus.cc:2454] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 994734068b204972bd8d4f53a6ef6985 in term 1.
I20260923 20:26:36.729410 1372 leader_election.cc:302] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63371bd0502d4026b100e925ecd73725, 994734068b204972bd8d4f53a6ef6985; no voters:
I20260923 20:26:36.729743 1636 raft_consensus.cc:2781] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:36.730271 1636 raft_consensus.cc:686] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 1 LEADER]: Becoming Leader. State: Replica: 994734068b204972bd8d4f53a6ef6985, State: Running, Role: LEADER
I20260923 20:26:36.730858 1636 consensus_queue.cc:237] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } }
I20260923 20:26:36.735252 1165 catalog_manager.cc:5495] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 reported cstate change: term changed from 0 to 1, leader changed from <none> to 994734068b204972bd8d4f53a6ef6985 (127.14.127.194). New cstate: current_term: 1 leader_uuid: "994734068b204972bd8d4f53a6ef6985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "994734068b204972bd8d4f53a6ef6985" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 35501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20260923 20:26:36.768299 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.763167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.14.127.194:35501)
0923 20:26:36.763336 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.763340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.763360 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:26:36.763553 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.763557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.763648 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.763782 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:26:36.764147 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.764925 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.766263 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.766740 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.766832 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.767317 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:26:36.767790 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:26:36.767923 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.768046 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.768082 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:26:36.776743 1306 raft_consensus.cc:1261] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 1 FOLLOWER]: Refusing update from remote peer 994734068b204972bd8d4f53a6ef6985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:36.776743 1567 raft_consensus.cc:1261] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 1 FOLLOWER]: Refusing update from remote peer 994734068b204972bd8d4f53a6ef6985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:36.777783 1636 consensus_queue.cc:1035] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af092cbaaca4923bdd29df636462fd2" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:36.778102 1641 consensus_queue.cc:1035] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63371bd0502d4026b100e925ecd73725" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:36.803958 1165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:36.804232 1165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:36.807941 1165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20538364e14e4a9db6be14df6e34e98f: Sending DeleteTablet for 3 replicas of tablet cc2bbdf53f7541059ccc27344a68cb9e
I20260923 20:26:36.809146 1547 tablet_service.cc:1514] Processing DeleteTablet for tablet cc2bbdf53f7541059ccc27344a68cb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:36 UTC) from {username='slave'} at 127.0.0.1:47358
I20260923 20:26:36.809206 1417 tablet_service.cc:1514] Processing DeleteTablet for tablet cc2bbdf53f7541059ccc27344a68cb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:36 UTC) from {username='slave'} at 127.0.0.1:47512
I20260923 20:26:36.809484 1286 tablet_service.cc:1514] Processing DeleteTablet for tablet cc2bbdf53f7541059ccc27344a68cb9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:36 UTC) from {username='slave'} at 127.0.0.1:39148
I20260923 20:26:36.811442 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 1224
I20260923 20:26:36.821666 1656 tablet_replica.cc:330] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: stopping tablet replica
I20260923 20:26:36.822219 1656 raft_consensus.cc:2229] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:36.822840 1656 raft_consensus.cc:2258] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:36.825426 1656 ts_tablet_manager.cc:1905] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:36.834475 1656 ts_tablet_manager.cc:1918] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:36.834672 1656 log.cc:1196] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/cc2bbdf53f7541059ccc27344a68cb9e
I20260923 20:26:36.835275 1656 ts_tablet_manager.cc:1939] T cc2bbdf53f7541059ccc27344a68cb9e P 0af092cbaaca4923bdd29df636462fd2: Deleting consensus metadata
I20260923 20:26:36.837549 1149 catalog_manager.cc:4841] TS 0af092cbaaca4923bdd29df636462fd2 (127.14.127.193:38629): tablet cc2bbdf53f7541059ccc27344a68cb9e (table test-table [id=abe9b0c2692540df9ebb0d72c3582a03]) successfully deleted
I20260923 20:26:36.838522 1654 tablet_replica.cc:330] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: stopping tablet replica
I20260923 20:26:36.839068 1654 raft_consensus.cc:2229] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:26:36.839818 1654 raft_consensus.cc:2258] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:36.842156 1654 ts_tablet_manager.cc:1905] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:36.842857 1655 tablet_replica.cc:330] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: stopping tablet replica
I20260923 20:26:36.843282 1655 raft_consensus.cc:2229] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:36.843797 1655 raft_consensus.cc:2258] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:36.846287 1655 ts_tablet_manager.cc:1905] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:36.849136 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.813006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.14.127.193:38629)
0923 20:26:36.813253 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.813263 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.838521 (+ 25258us) server_negotiation.cc:407] Connection header received
0923 20:26:36.839088 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.839096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.839220 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.839306 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:36.839990 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.841083 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.843044 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.843684 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.843812 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.847098 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:36.847111 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:36.847140 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:36.847920 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:36.847982 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:36.848007 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:36.848100 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.848647 (+ 547us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.848769 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:26:36.850497 1346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:36.850474 1654 ts_tablet_manager.cc:1918] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:36.850718 1654 log.cc:1196] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/cc2bbdf53f7541059ccc27344a68cb9e
I20260923 20:26:36.851241 1654 ts_tablet_manager.cc:1939] T cc2bbdf53f7541059ccc27344a68cb9e P 994734068b204972bd8d4f53a6ef6985: Deleting consensus metadata
I20260923 20:26:36.853530 1151 catalog_manager.cc:4841] TS 994734068b204972bd8d4f53a6ef6985 (127.14.127.194:35501): tablet cc2bbdf53f7541059ccc27344a68cb9e (table test-table [id=abe9b0c2692540df9ebb0d72c3582a03]) successfully deleted
I20260923 20:26:36.853790 1655 ts_tablet_manager.cc:1918] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:36.853996 1655 log.cc:1196] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/cc2bbdf53f7541059ccc27344a68cb9e
I20260923 20:26:36.854704 1655 ts_tablet_manager.cc:1939] T cc2bbdf53f7541059ccc27344a68cb9e P 63371bd0502d4026b100e925ecd73725: Deleting consensus metadata
I20260923 20:26:36.856529 1151 catalog_manager.cc:4841] TS 63371bd0502d4026b100e925ecd73725 (127.14.127.195:39763): tablet cc2bbdf53f7541059ccc27344a68cb9e (table test-table [id=abe9b0c2692540df9ebb0d72c3582a03]) successfully deleted
I20260923 20:26:36.941452 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 1224
I20260923 20:26:36.966003 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 1355
I20260923 20:26:36.974045 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:36.966807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.14.127.194:35501)
0923 20:26:36.967023 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:36.967030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:36.967051 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:26:36.967255 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:36.967259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:36.967341 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:36.967423 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:36.967927 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.968719 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.970593 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:36.971223 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:36.971322 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:36.972469 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:36.972477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:36.972491 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:36.973111 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:36.973169 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:36.973190 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:36.973287 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:36.973467 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 20:26:36.973546 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:26:36.974833 1477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:37.087095 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 1355
I20260923 20:26:37.105618 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 1485
I20260923 20:26:37.112500 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:37.106502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.14.127.195:39763)
0923 20:26:37.106710 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:37.106715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:37.106790 (+ 75us) server_negotiation.cc:407] Connection header received
0923 20:26:37.106940 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:37.106944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:37.107038 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:37.107134 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:37.107467 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:37.108212 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:37.109421 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:37.109816 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:37.109904 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:37.111146 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:37.111155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:37.111170 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:37.111744 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:37.111790 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:37.111809 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:37.111942 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:37.112053 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 20:26:37.112170 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:26:37.113161 1607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:37.207939 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 1485
I20260923 20:26:37.225243 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 1134
I20260923 20:26:37.226296 1195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:37.335214 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 1134
2026-09-23T20:26:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2730 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:37 dist-test-slave-wmd0 krb5kdc[1664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:37 dist-test-slave-wmd0 krb5kdc[1664](info): set up 2 sockets
Sep 23 20:26:37 dist-test-slave-wmd0 krb5kdc[1664](info): commencing operation
krb5kdc: starting...
W20260923 20:26:38.455122 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:38 dist-test-slave-wmd0 krb5kdc[1664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:26:38.701856 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:33799
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:39.028661 1677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:39.029007 1677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:39.029177 1677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:39.036278 1677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:39.036434 1677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:39.036546 1677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:39.036660 1677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:39.036749 1677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:39.046329 1677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:33799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:39.047685 1677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:39.049018 1677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:39.060501 1683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:39.060694 1685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:39.060751 1682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:39.061713 1677 server_base.cc:1034] running on GCE node
I20260923 20:26:39.062394 1677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:39.063735 1677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:39.064931 1677 hybrid_clock.cc:648] HybridClock initialized: now 1790195199064884 us; error 43 us; skew 500 ppm
I20260923 20:26:39.065238 1677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:39 dist-test-slave-wmd0 krb5kdc[1664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195199, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:39.068465 1677 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:26:39.069865 1677 webserver.cc:460] Webserver started at http://127.14.127.254:37963/ using document root <none> and password file <none>
I20260923 20:26:39.070488 1677 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:39.070585 1677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:39.070901 1677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:39.073711 1677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "963d7f2b040f459f9469c92381fa279f"
format_stamp: "Formatted at 2026-09-23 20:26:39 on dist-test-slave-wmd0"
server_key: "3fa6b217b99fc55f3bc2e2bbd927888d"
server_key_iv: "5177fb895a78ab6e2d324158148c0357"
server_key_version: "encryptionkey@0"
I20260923 20:26:39.074498 1677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "963d7f2b040f459f9469c92381fa279f"
format_stamp: "Formatted at 2026-09-23 20:26:39 on dist-test-slave-wmd0"
server_key: "3fa6b217b99fc55f3bc2e2bbd927888d"
server_key_iv: "5177fb895a78ab6e2d324158148c0357"
server_key_version: "encryptionkey@0"
I20260923 20:26:39.079788 1677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 20:26:39.083248 1692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:39.084537 1677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:26:39.084702 1677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "963d7f2b040f459f9469c92381fa279f"
format_stamp: "Formatted at 2026-09-23 20:26:39 on dist-test-slave-wmd0"
server_key: "3fa6b217b99fc55f3bc2e2bbd927888d"
server_key_iv: "5177fb895a78ab6e2d324158148c0357"
server_key_version: "encryptionkey@0"
I20260923 20:26:39.084874 1677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:39.096972 1677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:39.101233 1677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:39.101517 1677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:39.119675 1677 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:33799
I20260923 20:26:39.119704 1743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:33799 every 8 connection(s)
I20260923 20:26:39.121042 1677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:39.124763 1744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:39.125366 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 1677
I20260923 20:26:39.125547 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:39.126008 14847 external_mini_cluster.cc:1428] Setting key 158c983d93b5ef7511e8c891f30da2a7
I20260923 20:26:39.136970 1744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f: Bootstrap starting.
Sep 23 20:26:39 dist-test-slave-wmd0 krb5kdc[1664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:39.140308 1744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:39.141355 1744 log.cc:824] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:39.144877 1744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f: No bootstrap required, opened a new log
I20260923 20:26:39.149530 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:39.127884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.14.127.254:33799)
0923 20:26:39.128662 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:39.128674 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:39.128714 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:26:39.130906 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:39.130929 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:39.131326 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:39.131713 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:39.133182 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:39.134493 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:39.135869 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:39.136547 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:39.136753 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:39.141074 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:39.141086 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:39.141100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:39.141143 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:39.144947 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:39.145679 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:39.145687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:39.145695 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:39.145901 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:39.146251 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:39.146257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:39.146261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:39.147045 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:39.147355 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:39.147671 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:26:39.148154 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":336,"threads_started":1}
I20260923 20:26:39.154356 1744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d7f2b040f459f9469c92381fa279f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 33799 } }
I20260923 20:26:39.154778 1744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:39.154860 1744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963d7f2b040f459f9469c92381fa279f, State: Initialized, Role: FOLLOWER
I20260923 20:26:39.155404 1744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d7f2b040f459f9469c92381fa279f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 33799 } }
I20260923 20:26:39.155617 1744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:39.155712 1744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:39.155839 1744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:39.157932 1744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d7f2b040f459f9469c92381fa279f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 33799 } }
I20260923 20:26:39.158528 1744 leader_election.cc:302] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 963d7f2b040f459f9469c92381fa279f; no voters:
I20260923 20:26:39.159175 1744 leader_election.cc:288] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:39.159279 1749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:39.160004 1749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [term 1 LEADER]: Becoming Leader. State: Replica: 963d7f2b040f459f9469c92381fa279f, State: Running, Role: LEADER
I20260923 20:26:39.160549 1749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d7f2b040f459f9469c92381fa279f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 33799 } }
I20260923 20:26:39.161257 1744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:39.163131 1750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "963d7f2b040f459f9469c92381fa279f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d7f2b040f459f9469c92381fa279f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 33799 } } }
I20260923 20:26:39.163445 1750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:39.164371 1755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:39.164222 1751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 963d7f2b040f459f9469c92381fa279f. Latest consensus state: current_term: 1 leader_uuid: "963d7f2b040f459f9469c92381fa279f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963d7f2b040f459f9469c92381fa279f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 33799 } } }
I20260923 20:26:39.164512 1751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:39.170017 1755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:39.179500 1755 catalog_manager.cc:1347] Generated new cluster ID: 9633c4c9365f47c19feb86bddec4b26a
I20260923 20:26:39.179618 1755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:39.204774 1755 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:39.206075 1755 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:39.212291 1755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f: Generated new TSK 0
I20260923 20:26:39.213297 1755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.127.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
Entry for principal kudu/127.14.127.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab.
I20260923 20:26:39.286013 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:33799
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:39.623404 1770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:39.623715 1770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:39.623888 1770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:39.631124 1770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:39.631278 1770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:39.631453 1770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:26:39.641144 1770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.193.keytab
--principal=kudu/127.14.127.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:33799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:39.642474 1770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:39.643808 1770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:39.657388 1775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:39.657418 1776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:39.657568 1778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:39.658280 1770 server_base.cc:1034] running on GCE node
I20260923 20:26:39.658910 1770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:39.659839 1770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:39.661052 1770 hybrid_clock.cc:648] HybridClock initialized: now 1790195199660992 us; error 59 us; skew 500 ppm
I20260923 20:26:39.661378 1770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:39 dist-test-slave-wmd0 krb5kdc[1664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195199, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:39.664645 1770 init.cc:377] Logged in from keytab as kudu/127.14.127.193@KRBTEST.COM (short username kudu)
I20260923 20:26:39.666147 1770 webserver.cc:460] Webserver started at http://127.14.127.193:45065/ using document root <none> and password file <none>
I20260923 20:26:39.666927 1770 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:39.667098 1770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:39.667423 1770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:39.670327 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "bb1b2ae57941401abf46ab2e185a6aa6"
format_stamp: "Formatted at 2026-09-23 20:26:39 on dist-test-slave-wmd0"
server_key: "8ae8491a2c60ad539ec8979ce5cd1249"
server_key_iv: "6fd85a730f895bcf94aed7f6540e6998"
server_key_version: "encryptionkey@0"
I20260923 20:26:39.671304 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "bb1b2ae57941401abf46ab2e185a6aa6"
format_stamp: "Formatted at 2026-09-23 20:26:39 on dist-test-slave-wmd0"
server_key: "8ae8491a2c60ad539ec8979ce5cd1249"
server_key_iv: "6fd85a730f895bcf94aed7f6540e6998"
server_key_version: "encryptionkey@0"
I20260923 20:26:39.676633 1770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:26:39.680151 1785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:39.681430 1770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 20:26:39.681608 1770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "bb1b2ae57941401abf46ab2e185a6aa6"
format_stamp: "Formatted at 2026-09-23 20:26:39 on dist-test-slave-wmd0"
server_key: "8ae8491a2c60ad539ec8979ce5cd1249"
server_key_iv: "6fd85a730f895bcf94aed7f6540e6998"
server_key_version: "encryptionkey@0"
I20260923 20:26:39.681805 1770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:39.694375 1770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:39.698184 1770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:39.698475 1770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:39.699491 1770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:39.700805 1770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:39.700910 1770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:39.701078 1770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:39.701159 1770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:39.734216 1770 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:40039
I20260923 20:26:39.734298 1897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:40039 every 8 connection(s)
I20260923 20:26:39.735684 1770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:26:39.743331 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 1770
I20260923 20:26:39.743525 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:26:39.743979 14847 external_mini_cluster.cc:1428] Setting key a0c26330064a8779b4e2bdb6cfe73863
Sep 23 20:26:39 dist-test-slave-wmd0 krb5kdc[1664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195199, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.193@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:39.758939 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:39.738188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:55989 (local address 127.14.127.254:33799)
0923 20:26:39.738678 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:39.738683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:39.739870 (+ 1187us) server_negotiation.cc:407] Connection header received
0923 20:26:39.742236 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:39.742260 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:39.742366 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:39.742496 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:39.744405 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:39.745382 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:39.747348 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:39.748151 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:39.748312 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:39.751881 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:39.751890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:39.751899 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:39.751947 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:39.755280 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:39.756330 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:39.756334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:39.756336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:39.756453 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:39.757012 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:39.757016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:39.757018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:39.757309 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:39.757431 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:39.758378 (+ 947us) server_negotiation.cc:299] Negotiation successful
0923 20:26:39.758570 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":221,"threads_started":1}
I20260923 20:26:39.760471 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:39.738855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:33799 (local address 127.14.127.193:55989)
0923 20:26:39.739686 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:39.739737 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:39.741944 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:39.742817 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:39.742828 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:39.743457 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:39.744210 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:39.744231 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:39.745572 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:39.745577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:39.747181 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:39.747196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:39.748378 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:39.748383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:39.748587 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:39.749317 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:39.749341 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:39.751595 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:39.755455 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:39.755465 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:39.755486 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:39.756190 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:39.756577 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:39.756584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:39.756591 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:39.756848 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:39.757446 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:39.757457 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:39.757943 (+ 486us) client_negotiation.cc:769] Sending connection context
0923 20:26:39.758276 (+ 333us) client_negotiation.cc:240] Negotiation successful
0923 20:26:39.758939 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":282,"threads_started":1}
I20260923 20:26:39.762233 1898 heartbeater.cc:347] Connected to a master server at 127.14.127.254:33799
I20260923 20:26:39.762666 1898 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:39.763569 1898 heartbeater.cc:511] Master 127.14.127.254:33799 requested a full tablet report, sending...
I20260923 20:26:39.765914 1709 ts_manager.cc:194] Registered new tserver with Master: bb1b2ae57941401abf46ab2e185a6aa6 (127.14.127.193:40039)
I20260923 20:26:39.767529 1709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.193@KRBTEST.COM'} at 127.14.127.193:55989
WARNING: no policy specified for kudu/127.14.127.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
Entry for principal kudu/127.14.127.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab.
I20260923 20:26:39.806173 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:33799
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:40.133742 1904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:40.134059 1904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:40.134225 1904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:40.141506 1904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:40.141655 1904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:40.141836 1904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:26:40.151583 1904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.194.keytab
--principal=kudu/127.14.127.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:33799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:40.152845 1904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:40.154232 1904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:40.167362 1909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:40.167351 1910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:40.167410 1912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:40.168509 1904 server_base.cc:1034] running on GCE node
I20260923 20:26:40.169131 1904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:40.170109 1904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:40.171315 1904 hybrid_clock.cc:648] HybridClock initialized: now 1790195200171244 us; error 57 us; skew 500 ppm
I20260923 20:26:40.171665 1904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:40 dist-test-slave-wmd0 krb5kdc[1664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195200, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:40.180900 1904 init.cc:377] Logged in from keytab as kudu/127.14.127.194@KRBTEST.COM (short username kudu)
I20260923 20:26:40.182468 1904 webserver.cc:460] Webserver started at http://127.14.127.194:39555/ using document root <none> and password file <none>
I20260923 20:26:40.183221 1904 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:40.183372 1904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:40.183692 1904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:40.186872 1904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "69c33050afa74ae5a228866b90e2d5ff"
format_stamp: "Formatted at 2026-09-23 20:26:40 on dist-test-slave-wmd0"
server_key: "61ce4d205829d3dba9e78c5678175729"
server_key_iv: "d4695bb3c1d09322e36391cf1c1e402f"
server_key_version: "encryptionkey@0"
I20260923 20:26:40.187647 1904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "69c33050afa74ae5a228866b90e2d5ff"
format_stamp: "Formatted at 2026-09-23 20:26:40 on dist-test-slave-wmd0"
server_key: "61ce4d205829d3dba9e78c5678175729"
server_key_iv: "d4695bb3c1d09322e36391cf1c1e402f"
server_key_version: "encryptionkey@0"
I20260923 20:26:40.192766 1904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 20:26:40.196204 1919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:40.197502 1904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:26:40.197687 1904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "69c33050afa74ae5a228866b90e2d5ff"
format_stamp: "Formatted at 2026-09-23 20:26:40 on dist-test-slave-wmd0"
server_key: "61ce4d205829d3dba9e78c5678175729"
server_key_iv: "d4695bb3c1d09322e36391cf1c1e402f"
server_key_version: "encryptionkey@0"
I20260923 20:26:40.197898 1904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:40.213104 1904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:40.217437 1904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:40.217732 1904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:40.218717 1904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:40.220006 1904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:40.220135 1904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:40.220302 1904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:40.220376 1904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:40.253181 1904 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:33583
I20260923 20:26:40.253226 2031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:33583 every 8 connection(s)
I20260923 20:26:40.254483 1904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:26:40.260141 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 1904
I20260923 20:26:40.260335 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:26:40.260783 14847 external_mini_cluster.cc:1428] Setting key 4be4670a7203f9f183cda67c523d7d03
Sep 23 20:26:40 dist-test-slave-wmd0 krb5kdc[1664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195200, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.194@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:40.275915 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.257092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:47169 (local address 127.14.127.254:33799)
0923 20:26:40.257327 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:40.257333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:40.258459 (+ 1126us) server_negotiation.cc:407] Connection header received
0923 20:26:40.260352 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:40.260358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:40.260457 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:40.260573 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:40.262642 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.263660 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.265183 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.265937 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.266075 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.269454 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:40.269463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:40.269471 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:40.269517 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:40.272588 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:40.273341 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:40.273350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:40.273357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:40.273512 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:40.273916 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:40.273924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:40.273930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:40.274320 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:40.274469 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:40.275221 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 20:26:40.275481 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:26:40.276571 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.257586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:33799 (local address 127.14.127.194:47169)
0923 20:26:40.258288 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:40.258335 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:40.260067 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:40.260822 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:40.260830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:40.261655 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:40.262431 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.262457 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.263824 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.263828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.265033 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.265045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.266140 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.266144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.266337 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.267062 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:40.267085 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:40.269214 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:40.272722 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:40.272728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:40.272753 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:40.273216 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:40.273631 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:40.273634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:40.273636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:40.273812 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:40.274438 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:40.274443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:40.274854 (+ 411us) client_negotiation.cc:769] Sending connection context
0923 20:26:40.275081 (+ 227us) client_negotiation.cc:240] Negotiation successful
0923 20:26:40.275470 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260923 20:26:40.278020 2032 heartbeater.cc:347] Connected to a master server at 127.14.127.254:33799
I20260923 20:26:40.278394 2032 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:40.279191 2032 heartbeater.cc:511] Master 127.14.127.254:33799 requested a full tablet report, sending...
I20260923 20:26:40.280658 1709 ts_manager.cc:194] Registered new tserver with Master: 69c33050afa74ae5a228866b90e2d5ff (127.14.127.194:33583)
I20260923 20:26:40.281467 1709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.194@KRBTEST.COM'} at 127.14.127.194:47169
WARNING: no policy specified for kudu/127.14.127.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
Entry for principal kudu/127.14.127.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab.
I20260923 20:26:40.317574 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:33799
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:40.641209 2037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:40.641476 2037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:40.641570 2037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:40.648622 2037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:40.648726 2037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:40.648839 2037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:26:40.658368 2037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.195.keytab
--principal=kudu/127.14.127.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:33799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:40.659646 2037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:40.660867 2037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:40.673750 2042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:40.673663 2043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:40.673702 2045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:40.674925 2037 server_base.cc:1034] running on GCE node
I20260923 20:26:40.675590 2037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:40.676501 2037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:40.677709 2037 hybrid_clock.cc:648] HybridClock initialized: now 1790195200677661 us; error 36 us; skew 500 ppm
I20260923 20:26:40.678061 2037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:40 dist-test-slave-wmd0 krb5kdc[1664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195200, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:40.685408 2037 init.cc:377] Logged in from keytab as kudu/127.14.127.195@KRBTEST.COM (short username kudu)
I20260923 20:26:40.687084 2037 webserver.cc:460] Webserver started at http://127.14.127.195:45611/ using document root <none> and password file <none>
I20260923 20:26:40.687837 2037 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:40.687990 2037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:40.688313 2037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:40.691413 2037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "a47c998fe642400b8d2442e1a179815c"
format_stamp: "Formatted at 2026-09-23 20:26:40 on dist-test-slave-wmd0"
server_key: "95c8f1e4f620bf313daa8a1647326a61"
server_key_iv: "934a9d8ce3ac4aeed5754295fbbf89c4"
server_key_version: "encryptionkey@0"
I20260923 20:26:40.692173 2037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "a47c998fe642400b8d2442e1a179815c"
format_stamp: "Formatted at 2026-09-23 20:26:40 on dist-test-slave-wmd0"
server_key: "95c8f1e4f620bf313daa8a1647326a61"
server_key_iv: "934a9d8ce3ac4aeed5754295fbbf89c4"
server_key_version: "encryptionkey@0"
I20260923 20:26:40.697175 2037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:26:40.700513 2052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:40.701865 2037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:26:40.702052 2037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "a47c998fe642400b8d2442e1a179815c"
format_stamp: "Formatted at 2026-09-23 20:26:40 on dist-test-slave-wmd0"
server_key: "95c8f1e4f620bf313daa8a1647326a61"
server_key_iv: "934a9d8ce3ac4aeed5754295fbbf89c4"
server_key_version: "encryptionkey@0"
I20260923 20:26:40.702244 2037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:40.723341 2037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:40.727309 2037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:40.727596 2037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:40.728569 2037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:40.730309 2037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:40.730384 2037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:40.730527 2037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:40.730577 2037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:40.766250 2037 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:39997
I20260923 20:26:40.766330 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:39997 every 8 connection(s)
I20260923 20:26:40.767619 2037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:26:40.771584 1898 heartbeater.cc:503] Master 127.14.127.254:33799 was elected leader, sending a full tablet report...
I20260923 20:26:40.773038 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 2037
I20260923 20:26:40.773226 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:26:40.773774 14847 external_mini_cluster.cc:1428] Setting key bfe2dbcedc0a951b1780a03c6d18404b
Sep 23 20:26:40 dist-test-slave-wmd0 krb5kdc[1664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195200, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.195@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:26:40.795635 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.770198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:52143 (local address 127.14.127.254:33799)
0923 20:26:40.770387 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:40.770392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:40.775322 (+ 4930us) server_negotiation.cc:407] Connection header received
0923 20:26:40.775436 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:40.775442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:40.775537 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:40.775664 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:40.777625 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.778513 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.781115 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.781545 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.781713 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.787277 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:40.787283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:40.787288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:40.787334 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:40.791429 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:40.792459 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:40.792466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:40.792472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:40.792639 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:40.793275 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:40.793279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:40.793283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:40.793735 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:40.793902 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:40.794943 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0923 20:26:40.795240 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:26:40.796991 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.770753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:33799 (local address 127.14.127.195:52143)
0923 20:26:40.771576 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:40.771660 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:40.774640 (+ 2980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:40.776044 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:40.776058 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:40.776654 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:40.777442 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.777473 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.778687 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.778694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.780963 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.780981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.781817 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.781825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.782165 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.783370 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:26:40.783411 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:40.786983 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:40.791606 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:40.791617 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:40.791634 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:40.792310 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:40.792794 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:26:40.792799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:26:40.792802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:26:40.793114 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:26:40.793912 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:40.793924 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:40.794422 (+ 498us) client_negotiation.cc:769] Sending connection context
0923 20:26:40.794808 (+ 386us) client_negotiation.cc:240] Negotiation successful
0923 20:26:40.795447 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":336,"threads_started":1}
I20260923 20:26:40.798677 2165 heartbeater.cc:347] Connected to a master server at 127.14.127.254:33799
I20260923 20:26:40.799037 2165 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:40.800025 2165 heartbeater.cc:511] Master 127.14.127.254:33799 requested a full tablet report, sending...
I20260923 20:26:40.801733 1709 ts_manager.cc:194] Registered new tserver with Master: a47c998fe642400b8d2442e1a179815c (127.14.127.195:39997)
I20260923 20:26:40.802973 1709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.127.195@KRBTEST.COM'} at 127.14.127.195:52143
I20260923 20:26:40.804164 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:26:40.826038 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.810212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.14.127.254:33799)
0923 20:26:40.810431 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:40.810440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:40.810902 (+ 462us) server_negotiation.cc:407] Connection header received
0923 20:26:40.811108 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:40.811113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:40.811206 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:40.811331 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:40.813284 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.814573 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.816609 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.817375 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.817527 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.820582 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:40.820591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:40.820598 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:40.820645 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:40.823797 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:40.824406 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:40.824410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:40.824412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:40.824541 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:40.824865 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:40.824868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:40.824870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:40.825165 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:40.825345 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:40.825628 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 20:26:40.825784 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:26:40.829662 1709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:26:40.836539 1709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:26:40.859630 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.848810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:33583 (local address 127.0.0.1:55424)
0923 20:26:40.850018 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:40.850035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:40.850143 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:40.851531 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:40.851538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:40.851567 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:40.852026 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.852040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.854205 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.854210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.856735 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.856750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.858589 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.858627 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.858838 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.858898 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 20:26:40.859000 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 20:26:40.859178 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":76,"thread_start_us":969,"threads_started":1}
I20260923 20:26:40.860826 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.848281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:39997 (local address 127.0.0.1:50068)
0923 20:26:40.849031 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:40.849090 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:40.849295 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:40.851863 (+ 2568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:40.851870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:40.851897 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:40.852384 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.852397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.855459 (+ 3062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.855464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.857574 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.857589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.860016 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.860025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.860233 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.860265 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:26:40.860321 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 20:26:40.860443 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":23,"thread_start_us":1094,"threads_started":1}
I20260923 20:26:40.861136 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.849209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.14.127.194:33583)
0923 20:26:40.850870 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:40.850882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:40.850914 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:26:40.851022 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:40.851030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:40.851292 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:40.852248 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:40.852467 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.854045 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.856912 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.858377 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.860285 (+ 1908us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.860469 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:40.860536 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 20:26:40.860716 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1499,"thread_start_us":1410,"threads_started":1}
I20260923 20:26:40.861941 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.848524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.14.127.195:39997)
0923 20:26:40.851269 (+ 2745us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:40.851277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:40.851307 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:26:40.851398 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:40.851407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:40.851639 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:40.853044 (+ 1405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:40.853694 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.855299 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.857990 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.859784 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.861164 (+ 1380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.861344 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:40.861417 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 20:26:40.861569 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2588,"thread_start_us":2472,"threads_started":1}
I20260923 20:26:40.865350 1967 tablet_service.cc:1467] Processing CreateTablet for tablet e01af8c875b5488fa6718270e158612b (DEFAULT_TABLE table=test-table [id=93e1677ec3a643c1b62656c247db20d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:40.866194 2100 tablet_service.cc:1467] Processing CreateTablet for tablet e01af8c875b5488fa6718270e158612b (DEFAULT_TABLE table=test-table [id=93e1677ec3a643c1b62656c247db20d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:40.866888 1967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01af8c875b5488fa6718270e158612b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:40.867671 2100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01af8c875b5488fa6718270e158612b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:40.868669 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.850342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:40039 (local address 127.0.0.1:55154)
0923 20:26:40.859981 (+ 9639us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:40.860005 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:40.860094 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:40.862500 (+ 2406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:40.862504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:40.862530 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:40.862752 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.862763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.864233 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.864239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.865932 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.865949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.867958 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.867964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.868146 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.868178 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:26:40.868245 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 20:26:40.868380 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9532,"thread_start_us":12661,"threads_started":1}
I20260923 20:26:40.869684 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.850739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.14.127.193:40039)
0923 20:26:40.861540 (+ 10801us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:40.861550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:40.861576 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:26:40.861672 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:40.861682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:40.862003 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:40.862518 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:40.862933 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.864047 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.866207 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.867764 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.868969 (+ 1205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.869123 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:40.869187 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 20:26:40.869348 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10631,"thread_start_us":11369,"threads_started":1}
I20260923 20:26:40.873409 1833 tablet_service.cc:1467] Processing CreateTablet for tablet e01af8c875b5488fa6718270e158612b (DEFAULT_TABLE table=test-table [id=93e1677ec3a643c1b62656c247db20d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:40.874850 1833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01af8c875b5488fa6718270e158612b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:40.880584 2183 tablet_bootstrap.cc:492] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Bootstrap starting.
I20260923 20:26:40.882570 2184 tablet_bootstrap.cc:492] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Bootstrap starting.
I20260923 20:26:40.884330 2183 tablet_bootstrap.cc:654] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:40.885645 2183 log.cc:824] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:40.886006 2184 tablet_bootstrap.cc:654] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:40.887418 2184 log.cc:824] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:40.888942 2187 tablet_bootstrap.cc:492] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Bootstrap starting.
I20260923 20:26:40.890800 2183 tablet_bootstrap.cc:492] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: No bootstrap required, opened a new log
I20260923 20:26:40.891108 2183 ts_tablet_manager.cc:1397] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 20:26:40.891870 2184 tablet_bootstrap.cc:492] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: No bootstrap required, opened a new log
I20260923 20:26:40.892131 2184 ts_tablet_manager.cc:1397] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:26:40.892467 2187 tablet_bootstrap.cc:654] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:40.893416 2187 log.cc:824] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:40.895865 2187 tablet_bootstrap.cc:492] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: No bootstrap required, opened a new log
I20260923 20:26:40.896147 2187 ts_tablet_manager.cc:1397] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 20:26:40.902781 2184 raft_consensus.cc:348] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:40.902781 2183 raft_consensus.cc:348] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:40.903270 2184 raft_consensus.cc:374] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:40.903270 2183 raft_consensus.cc:374] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:40.903376 2183 raft_consensus.cc:729] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47c998fe642400b8d2442e1a179815c, State: Initialized, Role: FOLLOWER
I20260923 20:26:40.903376 2184 raft_consensus.cc:729] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c33050afa74ae5a228866b90e2d5ff, State: Initialized, Role: FOLLOWER
I20260923 20:26:40.904032 2183 consensus_queue.cc:260] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:40.904031 2184 consensus_queue.cc:260] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:40.904065 2187 raft_consensus.cc:348] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:40.904443 2187 raft_consensus.cc:374] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:40.904558 2187 raft_consensus.cc:729] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1b2ae57941401abf46ab2e185a6aa6, State: Initialized, Role: FOLLOWER
I20260923 20:26:40.905357 2165 heartbeater.cc:503] Master 127.14.127.254:33799 was elected leader, sending a full tablet report...
I20260923 20:26:40.906163 2183 ts_tablet_manager.cc:1428] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260923 20:26:40.906538 2184 ts_tablet_manager.cc:1428] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260923 20:26:40.906633 2032 heartbeater.cc:503] Master 127.14.127.254:33799 was elected leader, sending a full tablet report...
I20260923 20:26:40.906994 2187 consensus_queue.cc:260] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:40.908835 2187 ts_tablet_manager.cc:1428] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260923 20:26:40.987393 2191 raft_consensus.cc:482] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260923 20:26:40.987977 1899 tablet.cc:2369] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:26:40.987732 2191 raft_consensus.cc:504] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:40.989709 2191 leader_election.cc:288] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47c998fe642400b8d2442e1a179815c (127.14.127.195:39997), 69c33050afa74ae5a228866b90e2d5ff (127.14.127.194:33583)
I20260923 20:26:40.996969 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.990256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:58817 (local address 127.14.127.194:33583)
0923 20:26:40.990496 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:40.990506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:40.991528 (+ 1022us) server_negotiation.cc:407] Connection header received
0923 20:26:40.991738 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:40.991742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:40.991832 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:40.991922 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:40.992544 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.993241 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.995224 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.995985 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.996132 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.996221 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:40.996569 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 20:26:40.996702 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:26:40.997000 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.990246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:52757 (local address 127.14.127.195:39997)
0923 20:26:40.990465 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:40.990474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:40.991441 (+ 967us) server_negotiation.cc:407] Connection header received
0923 20:26:40.991633 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:40.991641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:40.991726 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:40.991805 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:40.992471 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.993241 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.995277 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.996012 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.996218 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.996290 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:40.996617 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 20:26:40.996750 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:26:40.997120 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.990245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:33583 (local address 127.14.127.193:58817)
0923 20:26:40.991426 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:40.991446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:40.991585 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:40.991978 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:40.991982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:40.992014 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:40.992352 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.992361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.993387 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.993392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.995066 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.995082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.996222 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.996229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.996418 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.996442 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:26:40.996516 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 20:26:40.996671 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":26,"thread_start_us":443,"threads_started":1}
I20260923 20:26:40.997505 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:40.990104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:39997 (local address 127.14.127.193:52757)
0923 20:26:40.991312 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:40.991329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:40.991456 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:40.991848 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:40.991854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:40.991883 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:40.992309 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.992318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.993371 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.993377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.995093 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:40.995105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:40.996271 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:40.996281 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:40.996438 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:40.996460 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:26:40.996566 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:26:40.996702 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":116,"thread_start_us":889,"threads_started":1}
I20260923 20:26:40.998178 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01af8c875b5488fa6718270e158612b" candidate_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c33050afa74ae5a228866b90e2d5ff" is_pre_election: true
I20260923 20:26:40.998517 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01af8c875b5488fa6718270e158612b" candidate_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47c998fe642400b8d2442e1a179815c" is_pre_election: true
I20260923 20:26:40.998732 1987 raft_consensus.cc:2454] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1b2ae57941401abf46ab2e185a6aa6 in term 0.
I20260923 20:26:40.999084 2120 raft_consensus.cc:2454] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1b2ae57941401abf46ab2e185a6aa6 in term 0.
I20260923 20:26:40.999404 1787 leader_election.cc:302] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69c33050afa74ae5a228866b90e2d5ff, bb1b2ae57941401abf46ab2e185a6aa6; no voters:
I20260923 20:26:40.999859 2191 raft_consensus.cc:2781] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:26:41.000003 2191 raft_consensus.cc:482] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:26:41.000113 2191 raft_consensus.cc:3037] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:41.002372 2191 raft_consensus.cc:504] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:41.003157 2191 leader_election.cc:288] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [CANDIDATE]: Term 1 election: Requested vote from peers a47c998fe642400b8d2442e1a179815c (127.14.127.195:39997), 69c33050afa74ae5a228866b90e2d5ff (127.14.127.194:33583)
I20260923 20:26:41.003605 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01af8c875b5488fa6718270e158612b" candidate_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47c998fe642400b8d2442e1a179815c"
I20260923 20:26:41.003654 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01af8c875b5488fa6718270e158612b" candidate_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69c33050afa74ae5a228866b90e2d5ff"
I20260923 20:26:41.003805 2120 raft_consensus.cc:3037] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:41.003844 1987 raft_consensus.cc:3037] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:41.006173 2120 raft_consensus.cc:2454] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1b2ae57941401abf46ab2e185a6aa6 in term 1.
I20260923 20:26:41.006220 1987 raft_consensus.cc:2454] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1b2ae57941401abf46ab2e185a6aa6 in term 1.
W20260923 20:26:41.006558 2033 tablet.cc:2369] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:26:41.006807 1787 leader_election.cc:302] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69c33050afa74ae5a228866b90e2d5ff, bb1b2ae57941401abf46ab2e185a6aa6; no voters:
I20260923 20:26:41.007170 2191 raft_consensus.cc:2781] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:41.007647 2191 raft_consensus.cc:686] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 1 LEADER]: Becoming Leader. State: Replica: bb1b2ae57941401abf46ab2e185a6aa6, State: Running, Role: LEADER
I20260923 20:26:41.008263 2191 consensus_queue.cc:237] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } }
I20260923 20:26:41.012828 1709 catalog_manager.cc:5495] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb1b2ae57941401abf46ab2e185a6aa6 (127.14.127.193). New cstate: current_term: 1 leader_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1b2ae57941401abf46ab2e185a6aa6" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 } health_report { overall_health: UNKNOWN } } }
W20260923 20:26:41.019163 2166 tablet.cc:2369] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:26:41.051911 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:41.046743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.14.127.193:40039)
0923 20:26:41.047001 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:41.047006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:41.047049 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:26:41.047257 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:41.047261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:41.047356 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:41.047480 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:26:41.047813 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:41.048536 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:41.049854 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:41.050261 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:41.050346 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:41.050805 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:26:41.051340 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:26:41.051449 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:41.051537 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 20:26:41.051608 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:26:41.059819 2120 raft_consensus.cc:1261] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 1 FOLLOWER]: Refusing update from remote peer bb1b2ae57941401abf46ab2e185a6aa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:41.059859 1987 raft_consensus.cc:1261] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 1 FOLLOWER]: Refusing update from remote peer bb1b2ae57941401abf46ab2e185a6aa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:41.060842 2194 consensus_queue.cc:1035] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69c33050afa74ae5a228866b90e2d5ff" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:41.061183 2191 consensus_queue.cc:1035] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47c998fe642400b8d2442e1a179815c" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:41.072185 2197 mvcc.cc:205] Tried to move back new op lower bound from 7332639543531515904 to 7332639543333859328. Current Snapshot: MvccSnapshot[applied={T|T < 7332639543531515904}]
I20260923 20:26:41.086212 1709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:41.086500 1709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:41.090167 1709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 963d7f2b040f459f9469c92381fa279f: Sending DeleteTablet for 3 replicas of tablet e01af8c875b5488fa6718270e158612b
I20260923 20:26:41.091408 1833 tablet_service.cc:1514] Processing DeleteTablet for tablet e01af8c875b5488fa6718270e158612b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:55154
I20260923 20:26:41.091400 2100 tablet_service.cc:1514] Processing DeleteTablet for tablet e01af8c875b5488fa6718270e158612b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:50068
I20260923 20:26:41.091399 1967 tablet_service.cc:1514] Processing DeleteTablet for tablet e01af8c875b5488fa6718270e158612b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:41 UTC) from {username='kudu', principal='kudu/127.14.127.254@KRBTEST.COM'} at 127.0.0.1:55424
I20260923 20:26:41.093847 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 1770
I20260923 20:26:41.103773 2207 tablet_replica.cc:330] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: stopping tablet replica
I20260923 20:26:41.104341 2207 raft_consensus.cc:2229] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:26:41.105149 2209 tablet_replica.cc:330] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: stopping tablet replica
I20260923 20:26:41.105547 2209 raft_consensus.cc:2229] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:41.106042 2209 raft_consensus.cc:2258] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:41.107179 2207 raft_consensus.cc:2258] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:41.108426 2209 ts_tablet_manager.cc:1905] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:41.109673 2207 ts_tablet_manager.cc:1905] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:41.111796 2208 tablet_replica.cc:330] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: stopping tablet replica
I20260923 20:26:41.112187 2208 raft_consensus.cc:2229] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:41.112654 2208 raft_consensus.cc:2258] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:41.115638 2209 ts_tablet_manager.cc:1918] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:41.115800 2209 log.cc:1196] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/e01af8c875b5488fa6718270e158612b
I20260923 20:26:41.116340 2209 ts_tablet_manager.cc:1939] T e01af8c875b5488fa6718270e158612b P a47c998fe642400b8d2442e1a179815c: Deleting consensus metadata
I20260923 20:26:41.118162 1695 catalog_manager.cc:4841] TS a47c998fe642400b8d2442e1a179815c (127.14.127.195:39997): tablet e01af8c875b5488fa6718270e158612b (table test-table [id=93e1677ec3a643c1b62656c247db20d7]) successfully deleted
I20260923 20:26:41.119042 2208 ts_tablet_manager.cc:1905] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:41.121229 2207 ts_tablet_manager.cc:1918] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:41.121393 2207 log.cc:1196] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/e01af8c875b5488fa6718270e158612b
I20260923 20:26:41.121920 2207 ts_tablet_manager.cc:1939] T e01af8c875b5488fa6718270e158612b P bb1b2ae57941401abf46ab2e185a6aa6: Deleting consensus metadata
I20260923 20:26:41.123648 1695 catalog_manager.cc:4841] TS bb1b2ae57941401abf46ab2e185a6aa6 (127.14.127.193:40039): tablet e01af8c875b5488fa6718270e158612b (table test-table [id=93e1677ec3a643c1b62656c247db20d7]) successfully deleted
I20260923 20:26:41.125631 2208 ts_tablet_manager.cc:1918] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 23 20:26:41 dist-test-slave-wmd0 krb5kdc[1664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.193@KRBTEST.COM
I20260923 20:26:41.125778 2208 log.cc:1196] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/e01af8c875b5488fa6718270e158612b
I20260923 20:26:41.126250 2208 ts_tablet_manager.cc:1939] T e01af8c875b5488fa6718270e158612b P 69c33050afa74ae5a228866b90e2d5ff: Deleting consensus metadata
I20260923 20:26:41.128192 1694 catalog_manager.cc:4841] TS 69c33050afa74ae5a228866b90e2d5ff (127.14.127.194:33583): tablet e01af8c875b5488fa6718270e158612b (table test-table [id=93e1677ec3a643c1b62656c247db20d7]) successfully deleted
I20260923 20:26:41.134296 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:41.095350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.14.127.193:40039)
0923 20:26:41.095520 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:41.095527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:41.109319 (+ 13792us) server_negotiation.cc:407] Connection header received
0923 20:26:41.109611 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:41.109616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:41.109711 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:41.109792 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:41.111828 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:41.113013 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:41.115143 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:41.115880 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:41.116005 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:41.127053 (+ 11048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:41.127067 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:41.127084 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:41.127130 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:41.131145 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:41.131977 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:41.131986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:41.131989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:41.132154 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:41.132572 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:41.132577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:41.132580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:41.133047 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:41.133332 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:41.133674 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 20:26:41.133911 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:26:41.135015 1893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:41.236104 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 1770
I20260923 20:26:41.255307 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 1904
Sep 23 20:26:41 dist-test-slave-wmd0 krb5kdc[1664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.194@KRBTEST.COM
I20260923 20:26:41.273779 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:41.256181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.14.127.194:33583)
0923 20:26:41.256353 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:41.256358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:41.256409 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:26:41.256608 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:41.256612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:41.256676 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:41.256769 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:41.258357 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:41.259566 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:41.261495 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:41.261921 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:41.262072 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:41.267007 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:41.267020 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:41.267038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:41.267081 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:41.270932 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:41.271665 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:41.271674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:41.271678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:41.271822 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:41.272226 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:41.272230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:41.272234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:41.272705 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:41.272967 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:41.273199 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 20:26:41.273427 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:26:41.274497 2027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:41.368396 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 1904
I20260923 20:26:41.385617 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2037
Sep 23 20:26:41 dist-test-slave-wmd0 krb5kdc[1664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.195@KRBTEST.COM
I20260923 20:26:41.401635 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:41.386433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.14.127.195:39997)
0923 20:26:41.386705 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:41.386712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:41.386815 (+ 103us) server_negotiation.cc:407] Connection header received
0923 20:26:41.387046 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:41.387050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:41.387119 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:41.387228 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:41.388582 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:41.389358 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:41.390845 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:41.391257 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:41.391402 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:41.395809 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:41.395825 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:41.395841 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:41.395874 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:41.398761 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:41.399524 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:41.399531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:41.399533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:41.399691 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:41.400084 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:41.400089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:41.400091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:41.400416 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:41.400729 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:41.401051 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 20:26:41.401312 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 20:26:41.402282 2160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:41.491926 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 2037
I20260923 20:26:41.508615 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 1677
I20260923 20:26:41.509753 1739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:41.619902 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 1677
2026-09-23T20:26:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4285 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T20:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:41Z Disabled control of system clock
I20260923 20:26:41.696449 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46311
--webserver_interface=127.14.127.254
--webserver_port=0
--builtin_ntp_servers=127.14.127.212:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:26:42.032609 2218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:42.032902 2218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:42.032984 2218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:42.040351 2218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:42.040465 2218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:42.040503 2218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:42.040553 2218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:42.040603 2218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:42.049926 2218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:46311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:42.051213 2218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:42.053022 2218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:42.064940 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:42.064989 2223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:42.064944 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:42.065872 2218 server_base.cc:1034] running on GCE node
I20260923 20:26:42.066723 2218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:42.068109 2218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:42.069308 2218 hybrid_clock.cc:648] HybridClock initialized: now 1790195202069255 us; error 54 us; skew 500 ppm
I20260923 20:26:42.069617 2218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:42.071964 2218 webserver.cc:460] Webserver started at http://127.14.127.254:42247/ using document root <none> and password file <none>
I20260923 20:26:42.072628 2218 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:42.072724 2218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:42.072988 2218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:42.075907 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "4da8203469004bbeaf583110a79ffd3e"
format_stamp: "Formatted at 2026-09-23 20:26:42 on dist-test-slave-wmd0"
server_key: "e1b003e005e632505ae502821d880d31"
server_key_iv: "4bb1b0eba9ba5758ab7dd3cd486f3b5f"
server_key_version: "encryptionkey@0"
I20260923 20:26:42.076659 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "4da8203469004bbeaf583110a79ffd3e"
format_stamp: "Formatted at 2026-09-23 20:26:42 on dist-test-slave-wmd0"
server_key: "e1b003e005e632505ae502821d880d31"
server_key_iv: "4bb1b0eba9ba5758ab7dd3cd486f3b5f"
server_key_version: "encryptionkey@0"
I20260923 20:26:42.081667 2218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:26:42.085182 2232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:42.086478 2218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:26:42.086687 2218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "4da8203469004bbeaf583110a79ffd3e"
format_stamp: "Formatted at 2026-09-23 20:26:42 on dist-test-slave-wmd0"
server_key: "e1b003e005e632505ae502821d880d31"
server_key_iv: "4bb1b0eba9ba5758ab7dd3cd486f3b5f"
server_key_version: "encryptionkey@0"
I20260923 20:26:42.086889 2218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:42.103276 2218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:42.104075 2218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:42.104328 2218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:42.123175 2218 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:46311
I20260923 20:26:42.123199 2283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:46311 every 8 connection(s)
I20260923 20:26:42.124600 2218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:42.128388 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:42.130136 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 2218
I20260923 20:26:42.130330 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:42.130851 14847 external_mini_cluster.cc:1428] Setting key cb9a29ca2fcc187a70cf28a837a2271b
I20260923 20:26:42.140200 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e: Bootstrap starting.
I20260923 20:26:42.143460 2284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:42.144600 2284 log.cc:824] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:42.144876 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:42.132811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.14.127.254:46311)
0923 20:26:42.133889 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:42.133900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:42.133947 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:26:42.135771 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:42.135797 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:42.136318 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:42.136753 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:42.137421 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:42.138861 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:42.140230 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:42.140998 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:42.141218 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:42.142455 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:42.142462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:42.142477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:42.143107 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:42.143196 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:42.143218 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:42.143346 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:42.143577 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 20:26:42.143907 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":388,"threads_started":1}
I20260923 20:26:42.147727 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e: No bootstrap required, opened a new log
I20260923 20:26:42.155176 2284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da8203469004bbeaf583110a79ffd3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46311 } }
I20260923 20:26:42.155560 2284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:42.155679 2284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da8203469004bbeaf583110a79ffd3e, State: Initialized, Role: FOLLOWER
I20260923 20:26:42.156311 2284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da8203469004bbeaf583110a79ffd3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46311 } }
I20260923 20:26:42.156543 2284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:42.156668 2284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:42.156838 2284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:42.159065 2284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da8203469004bbeaf583110a79ffd3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46311 } }
I20260923 20:26:42.159578 2284 leader_election.cc:302] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4da8203469004bbeaf583110a79ffd3e; no voters:
I20260923 20:26:42.160324 2284 leader_election.cc:288] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:42.160475 2289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:42.161252 2289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [term 1 LEADER]: Becoming Leader. State: Replica: 4da8203469004bbeaf583110a79ffd3e, State: Running, Role: LEADER
I20260923 20:26:42.161798 2289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da8203469004bbeaf583110a79ffd3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46311 } }
I20260923 20:26:42.162480 2284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:42.165494 2290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4da8203469004bbeaf583110a79ffd3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da8203469004bbeaf583110a79ffd3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46311 } } }
I20260923 20:26:42.165292 2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4da8203469004bbeaf583110a79ffd3e. Latest consensus state: current_term: 1 leader_uuid: "4da8203469004bbeaf583110a79ffd3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da8203469004bbeaf583110a79ffd3e" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 46311 } } }
I20260923 20:26:42.165860 2290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:42.165910 2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:42.168406 2295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:42.174068 2295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:42.182693 2295 catalog_manager.cc:1347] Generated new cluster ID: a83a1eccbaaf4d5d9c06f8b8d12d0ebc
I20260923 20:26:42.182811 2295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:42.212337 2295 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:42.214155 2295 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:42.224011 2295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e: Generated new TSK 0
I20260923 20:26:42.224848 2295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:42.230406 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.193:0
--local_ip_for_outbound_sockets=127.14.127.193
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46311
--builtin_ntp_servers=127.14.127.212:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:26:42.577077 2308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:42.577373 2308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:42.577461 2308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:42.584796 2308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:42.584900 2308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:42.585014 2308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.193
I20260923 20:26:42.594520 2308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.127.193
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.193
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:42.595863 2308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:42.597641 2308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:42.612027 2313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:42.611967 2316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:42.611953 2314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:42.612884 2308 server_base.cc:1034] running on GCE node
I20260923 20:26:42.613523 2308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:42.614500 2308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:42.615756 2308 hybrid_clock.cc:648] HybridClock initialized: now 1790195202615695 us; error 53 us; skew 500 ppm
I20260923 20:26:42.616132 2308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:42.618584 2308 webserver.cc:460] Webserver started at http://127.14.127.193:44581/ using document root <none> and password file <none>
I20260923 20:26:42.619345 2308 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:42.619470 2308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:42.619794 2308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:42.622722 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/data/instance:
uuid: "471dc80bd2d3420aababd11ba5d59f33"
format_stamp: "Formatted at 2026-09-23 20:26:42 on dist-test-slave-wmd0"
server_key: "90a5c030a92052cda95875ca4d88b20a"
server_key_iv: "034483c2e4451f0ca00b21b203c81d72"
server_key_version: "encryptionkey@0"
I20260923 20:26:42.623521 2308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance:
uuid: "471dc80bd2d3420aababd11ba5d59f33"
format_stamp: "Formatted at 2026-09-23 20:26:42 on dist-test-slave-wmd0"
server_key: "90a5c030a92052cda95875ca4d88b20a"
server_key_iv: "034483c2e4451f0ca00b21b203c81d72"
server_key_version: "encryptionkey@0"
I20260923 20:26:42.628624 2308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:26:42.632139 2322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:42.633432 2308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:26:42.633611 2308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/wal
uuid: "471dc80bd2d3420aababd11ba5d59f33"
format_stamp: "Formatted at 2026-09-23 20:26:42 on dist-test-slave-wmd0"
server_key: "90a5c030a92052cda95875ca4d88b20a"
server_key_iv: "034483c2e4451f0ca00b21b203c81d72"
server_key_version: "encryptionkey@0"
I20260923 20:26:42.633836 2308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:42.667047 2308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:42.668011 2308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:42.668337 2308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:42.669523 2308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:42.671164 2308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:42.671298 2308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:42.671424 2308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:42.671512 2308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:42.704655 2308 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.193:36053
I20260923 20:26:42.704743 2434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.193:36053 every 8 connection(s)
I20260923 20:26:42.705984 2308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/data/info.pb
I20260923 20:26:42.710325 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 2308
I20260923 20:26:42.710508 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/wal/instance
I20260923 20:26:42.711089 14847 external_mini_cluster.cc:1428] Setting key ba8fea1a830a78e783725fe067a29820
I20260923 20:26:42.714020 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.194:0
--local_ip_for_outbound_sockets=127.14.127.194
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46311
--builtin_ntp_servers=127.14.127.212:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:26:42.720368 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:42.708540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.193:51105 (local address 127.14.127.254:46311)
0923 20:26:42.708978 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:42.708983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:42.710354 (+ 1371us) server_negotiation.cc:407] Connection header received
0923 20:26:42.712547 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:42.712553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:42.712653 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:42.712771 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:42.714271 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:42.715500 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:42.717064 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:42.717765 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:42.717907 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:42.719347 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:42.719356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:42.719364 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:42.719389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:42.719442 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:42.719460 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:42.719553 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:42.719832 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 20:26:42.719941 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
I20260923 20:26:42.721256 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:42.709173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46311 (local address 127.14.127.193:51105)
0923 20:26:42.710129 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:42.710180 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:42.712287 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:42.713056 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:42.713068 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:42.713186 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:42.713991 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:42.714020 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:42.715665 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:42.715670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:42.716900 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:42.716917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:42.718017 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:42.718023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:42.718229 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:42.719031 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:42.719033 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:42.719074 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:42.719154 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:42.719606 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:42.719616 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:42.719623 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 20:26:42.719791 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 20:26:42.720102 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":258,"threads_started":1}
I20260923 20:26:42.723073 2435 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46311
I20260923 20:26:42.723394 2435 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:42.724273 2435 heartbeater.cc:511] Master 127.14.127.254:46311 requested a full tablet report, sending...
I20260923 20:26:42.726527 2249 ts_manager.cc:194] Registered new tserver with Master: 471dc80bd2d3420aababd11ba5d59f33 (127.14.127.193:36053)
I20260923 20:26:42.728018 2249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.193:51105
W20260923 20:26:43.041445 2439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:43.041726 2439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:43.041812 2439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:43.048904 2439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:43.049000 2439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:43.049197 2439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.194
I20260923 20:26:43.058081 2439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.127.194
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.194
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:43.059365 2439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:43.061203 2439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:43.074579 2445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:43.074720 2444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:43.074697 2447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:43.075397 2439 server_base.cc:1034] running on GCE node
I20260923 20:26:43.076050 2439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:43.077092 2439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:43.078356 2439 hybrid_clock.cc:648] HybridClock initialized: now 1790195203078308 us; error 54 us; skew 500 ppm
I20260923 20:26:43.078766 2439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:43.081125 2439 webserver.cc:460] Webserver started at http://127.14.127.194:43301/ using document root <none> and password file <none>
I20260923 20:26:43.081892 2439 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:43.082031 2439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:43.082361 2439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:43.085570 2439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/data/instance:
uuid: "edf46ba8e80c406c989ce26488ae6b5a"
format_stamp: "Formatted at 2026-09-23 20:26:43 on dist-test-slave-wmd0"
server_key: "edf75a247c10c85c01e6fea9ed0b94c0"
server_key_iv: "ee360a389e95c3d8672167162f138f9e"
server_key_version: "encryptionkey@0"
I20260923 20:26:43.086398 2439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance:
uuid: "edf46ba8e80c406c989ce26488ae6b5a"
format_stamp: "Formatted at 2026-09-23 20:26:43 on dist-test-slave-wmd0"
server_key: "edf75a247c10c85c01e6fea9ed0b94c0"
server_key_iv: "ee360a389e95c3d8672167162f138f9e"
server_key_version: "encryptionkey@0"
I20260923 20:26:43.091967 2439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260923 20:26:43.095698 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:43.097076 2439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:43.097270 2439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/wal
uuid: "edf46ba8e80c406c989ce26488ae6b5a"
format_stamp: "Formatted at 2026-09-23 20:26:43 on dist-test-slave-wmd0"
server_key: "edf75a247c10c85c01e6fea9ed0b94c0"
server_key_iv: "ee360a389e95c3d8672167162f138f9e"
server_key_version: "encryptionkey@0"
I20260923 20:26:43.097503 2439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:43.112639 2439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:43.113538 2439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:43.113860 2439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:43.114933 2439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:43.116461 2439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:43.116592 2439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:43.116737 2439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:43.116833 2439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:43.153113 2439 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.194:45437
I20260923 20:26:43.153158 2565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.194:45437 every 8 connection(s)
I20260923 20:26:43.154536 2439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/data/info.pb
I20260923 20:26:43.161540 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 2439
I20260923 20:26:43.161747 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/wal/instance
I20260923 20:26:43.162351 14847 external_mini_cluster.cc:1428] Setting key c7dd700e563ae2762bccd483c721beea
I20260923 20:26:43.165534 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.127.195:0
--local_ip_for_outbound_sockets=127.14.127.195
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46311
--builtin_ntp_servers=127.14.127.212:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:26:43.168530 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.157190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.194:48045 (local address 127.14.127.254:46311)
0923 20:26:43.157425 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.157432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.158996 (+ 1564us) server_negotiation.cc:407] Connection header received
0923 20:26:43.160755 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.160760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.160831 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.160926 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:43.162093 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.163266 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.165302 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.166056 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.166213 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.167575 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:43.167585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:43.167593 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:43.167620 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:43.167665 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:43.167685 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:43.167789 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.168002 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.168121 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:26:43.169486 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.157870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46311 (local address 127.14.127.194:48045)
0923 20:26:43.158837 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:43.158879 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:43.160497 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:43.161165 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:43.161183 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:43.161269 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:43.161915 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.161934 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.163469 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.163475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.165146 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.165159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.166262 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.166267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.166466 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.167262 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:43.167264 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:43.167303 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:43.167377 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:43.167790 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:43.167798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:43.167803 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:26:43.167941 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 20:26:43.168342 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":281,"threads_started":1}
I20260923 20:26:43.170938 2566 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46311
I20260923 20:26:43.171284 2566 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:43.172228 2566 heartbeater.cc:511] Master 127.14.127.254:46311 requested a full tablet report, sending...
I20260923 20:26:43.173799 2249 ts_manager.cc:194] Registered new tserver with Master: edf46ba8e80c406c989ce26488ae6b5a (127.14.127.194:45437)
I20260923 20:26:43.174652 2249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.194:48045
W20260923 20:26:43.496150 2569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:43.496418 2569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:43.496502 2569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:43.504012 2569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:43.504122 2569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:43.504293 2569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.127.195
I20260923 20:26:43.513530 2569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.127.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.127.195
--webserver_port=0
--tserver_master_addrs=127.14.127.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.127.195
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:43.514858 2569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:43.516637 2569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:43.530011 2574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:43.529989 2575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:43.530092 2577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:43.530814 2569 server_base.cc:1034] running on GCE node
I20260923 20:26:43.531427 2569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:43.532387 2569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:43.533635 2569 hybrid_clock.cc:648] HybridClock initialized: now 1790195203533591 us; error 58 us; skew 500 ppm
I20260923 20:26:43.533964 2569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:26:43.536314 2569 webserver.cc:460] Webserver started at http://127.14.127.195:43837/ using document root <none> and password file <none>
I20260923 20:26:43.537024 2569 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:43.537164 2569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:43.537573 2569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:43.540529 2569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/data/instance:
uuid: "8e60c7c3c36e4304ac3e33f9aa323860"
format_stamp: "Formatted at 2026-09-23 20:26:43 on dist-test-slave-wmd0"
server_key: "68f1d77e4eff1cd5ffd6b769dca6d7a1"
server_key_iv: "ca22a51b11e5b0d3178b3f8a3464a64c"
server_key_version: "encryptionkey@0"
I20260923 20:26:43.541358 2569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance:
uuid: "8e60c7c3c36e4304ac3e33f9aa323860"
format_stamp: "Formatted at 2026-09-23 20:26:43 on dist-test-slave-wmd0"
server_key: "68f1d77e4eff1cd5ffd6b769dca6d7a1"
server_key_iv: "ca22a51b11e5b0d3178b3f8a3464a64c"
server_key_version: "encryptionkey@0"
I20260923 20:26:43.546464 2569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:26:43.549929 2583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:43.551312 2569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:26:43.551496 2569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/wal
uuid: "8e60c7c3c36e4304ac3e33f9aa323860"
format_stamp: "Formatted at 2026-09-23 20:26:43 on dist-test-slave-wmd0"
server_key: "68f1d77e4eff1cd5ffd6b769dca6d7a1"
server_key_iv: "ca22a51b11e5b0d3178b3f8a3464a64c"
server_key_version: "encryptionkey@0"
I20260923 20:26:43.551738 2569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:43.569504 2569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:43.570398 2569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:43.570803 2569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:43.571877 2569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:26:43.573572 2569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:26:43.573809 2569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:43.573961 2569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:26:43.574064 2569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:43.606719 2569 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.195:40435
I20260923 20:26:43.606765 2695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.195:40435 every 8 connection(s)
I20260923 20:26:43.608072 2569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/data/info.pb
I20260923 20:26:43.613142 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 2569
I20260923 20:26:43.613333 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/wal/instance
I20260923 20:26:43.613750 14847 external_mini_cluster.cc:1428] Setting key 42dbfd5464d536ffd5fc9d43f68cfd8b
I20260923 20:26:43.620782 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.610714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:53377 (local address 127.14.127.254:46311)
0923 20:26:43.610927 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.610935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.612322 (+ 1387us) server_negotiation.cc:407] Connection header received
0923 20:26:43.614264 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.614269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.614336 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.614422 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:43.615717 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.616762 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.618253 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.618709 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.618802 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.620141 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:43.620145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:43.620148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:43.620165 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:43.620191 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:43.620201 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:43.620267 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.620464 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.620533 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:26:43.621737 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.611310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:46311 (local address 127.14.127.195:53377)
0923 20:26:43.612168 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:43.612208 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:43.613970 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:43.614710 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:43.614719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:43.614828 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:26:43.615535 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.615561 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.616895 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.616899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.618130 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.618142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.618883 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.618887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.619098 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.619833 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:26:43.619835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:26:43.619874 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:26:43.619947 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:26:43.620284 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:26:43.620291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:26:43.620297 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:26:43.620442 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 20:26:43.620733 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":254,"threads_started":1}
I20260923 20:26:43.623169 2696 heartbeater.cc:347] Connected to a master server at 127.14.127.254:46311
I20260923 20:26:43.623472 2696 heartbeater.cc:464] Registering TS with master...
I20260923 20:26:43.624182 2696 heartbeater.cc:511] Master 127.14.127.254:46311 requested a full tablet report, sending...
I20260923 20:26:43.625545 2249 ts_manager.cc:194] Registered new tserver with Master: 8e60c7c3c36e4304ac3e33f9aa323860 (127.14.127.195:40435)
I20260923 20:26:43.626282 2249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.127.195:53377
I20260923 20:26:43.628825 14847 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:26:43.638660 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.632712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.14.127.254:46311)
0923 20:26:43.632867 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.632872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.633317 (+ 445us) server_negotiation.cc:407] Connection header received
0923 20:26:43.633507 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.633512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.633594 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.633697 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:43.634183 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.634989 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.636287 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.636690 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.636817 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.637932 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:43.637937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:43.637940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:43.637957 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:43.637984 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:43.637996 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:43.638098 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.638245 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.638340 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:26:43.642300 2249 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:26:43.648942 2249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:26:43.681974 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.660831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.14.127.194:45437)
0923 20:26:43.672719 (+ 11888us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.672729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.672753 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:26:43.672830 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.672836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.673060 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.674064 (+ 1004us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:43.674287 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.675891 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.679734 (+ 3843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.680757 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.680952 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.681189 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.681421 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.681603 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11730,"thread_start_us":11627,"threads_started":1}
I20260923 20:26:43.682011 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.660748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:45437 (local address 127.0.0.1:45942)
0923 20:26:43.671503 (+ 10755us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:43.671567 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:43.671811 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:43.673295 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:43.673304 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:43.673338 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:43.673839 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.673851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.676052 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.676061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.679297 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.679312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.680960 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.680966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.681173 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.681231 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 20:26:43.681376 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 20:26:43.681568 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10384,"mutex_wait_us":399,"thread_start_us":11400,"threads_started":1}
I20260923 20:26:43.682214 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.660853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.195:40435 (local address 127.0.0.1:43246)
0923 20:26:43.672407 (+ 11554us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:43.672426 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:43.672539 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:43.673535 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:43.673539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:43.673559 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:43.674128 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.674140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.676059 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.676064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.679297 (+ 3233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.679312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.681336 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.681346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.681529 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.681559 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:26:43.681655 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 20:26:43.681829 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11426,"mutex_wait_us":273,"thread_start_us":10087,"threads_started":1}
I20260923 20:26:43.682981 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.661186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.14.127.195:40435)
0923 20:26:43.670902 (+ 9716us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.670914 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.672679 (+ 1765us) server_negotiation.cc:407] Connection header received
0923 20:26:43.672796 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.672804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.673060 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.673258 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:43.674322 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.675912 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.679575 (+ 3663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.680920 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.681094 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.681294 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.682440 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.682661 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9524,"thread_start_us":9365,"threads_started":1}
I20260923 20:26:43.686373 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.676175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41018)
0923 20:26:43.677569 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:43.677595 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:43.677758 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:43.679288 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:43.679293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:43.679317 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:43.679751 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.679763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.681571 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.681577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.683674 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.683692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.685661 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.685668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.685835 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.685867 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:26:43.685965 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 20:26:43.686087 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"thread_start_us":1011,"threads_started":1}
I20260923 20:26:43.686484 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.672330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.14.127.193:36053)
0923 20:26:43.678276 (+ 5946us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.678302 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.678327 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:26:43.678474 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.678482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.678786 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.678997 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:43.679945 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.681406 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.683891 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.685431 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.685587 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.685793 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.686029 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.686183 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5799,"thread_start_us":5684,"threads_started":1}
I20260923 20:26:43.686524 2631 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb6071c11dd46a1b0faf2cb470efb77 (DEFAULT_TABLE table=test-table [id=14d47b59ca024a9f935040c6811c5da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:43.686519 2501 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb6071c11dd46a1b0faf2cb470efb77 (DEFAULT_TABLE table=test-table [id=14d47b59ca024a9f935040c6811c5da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:43.688139 2631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb6071c11dd46a1b0faf2cb470efb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:43.688346 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb6071c11dd46a1b0faf2cb470efb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:43.694530 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb6071c11dd46a1b0faf2cb470efb77 (DEFAULT_TABLE table=test-table [id=14d47b59ca024a9f935040c6811c5da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:26:43.696130 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb6071c11dd46a1b0faf2cb470efb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:43.699824 2714 tablet_bootstrap.cc:492] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: Bootstrap starting.
I20260923 20:26:43.703296 2714 tablet_bootstrap.cc:654] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:43.704638 2714 log.cc:824] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:43.708298 2715 tablet_bootstrap.cc:492] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: Bootstrap starting.
I20260923 20:26:43.713800 2715 tablet_bootstrap.cc:654] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:43.714932 2714 tablet_bootstrap.cc:492] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: No bootstrap required, opened a new log
I20260923 20:26:43.715261 2714 ts_tablet_manager.cc:1397] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260923 20:26:43.715584 2715 log.cc:824] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:43.719539 2717 tablet_bootstrap.cc:492] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: Bootstrap starting.
I20260923 20:26:43.722852 2717 tablet_bootstrap.cc:654] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:43.724299 2714 raft_consensus.cc:348] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.724690 2714 raft_consensus.cc:374] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:43.724850 2714 raft_consensus.cc:729] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf46ba8e80c406c989ce26488ae6b5a, State: Initialized, Role: FOLLOWER
I20260923 20:26:43.725109 2717 log.cc:824] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:43.725533 2715 tablet_bootstrap.cc:492] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: No bootstrap required, opened a new log
I20260923 20:26:43.725462 2714 consensus_queue.cc:260] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.725912 2715 ts_tablet_manager.cc:1397] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260923 20:26:43.726670 2566 heartbeater.cc:503] Master 127.14.127.254:46311 was elected leader, sending a full tablet report...
I20260923 20:26:43.727285 2714 ts_tablet_manager.cc:1428] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260923 20:26:43.729832 2717 tablet_bootstrap.cc:492] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: No bootstrap required, opened a new log
I20260923 20:26:43.730139 2717 ts_tablet_manager.cc:1397] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260923 20:26:43.731580 2435 heartbeater.cc:503] Master 127.14.127.254:46311 was elected leader, sending a full tablet report...
I20260923 20:26:43.736768 2715 raft_consensus.cc:348] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.737250 2715 raft_consensus.cc:374] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:43.737360 2715 raft_consensus.cc:729] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e60c7c3c36e4304ac3e33f9aa323860, State: Initialized, Role: FOLLOWER
I20260923 20:26:43.738019 2715 consensus_queue.cc:260] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.739166 2696 heartbeater.cc:503] Master 127.14.127.254:46311 was elected leader, sending a full tablet report...
I20260923 20:26:43.739959 2715 ts_tablet_manager.cc:1428] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 20:26:43.740790 2717 raft_consensus.cc:348] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.741163 2717 raft_consensus.cc:374] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:43.741262 2717 raft_consensus.cc:729] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471dc80bd2d3420aababd11ba5d59f33, State: Initialized, Role: FOLLOWER
I20260923 20:26:43.741891 2717 consensus_queue.cc:260] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.743748 2717 ts_tablet_manager.cc:1428] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: Time spent starting tablet: real 0.013s user 0.008s sys 0.007s
I20260923 20:26:43.764601 2721 raft_consensus.cc:482] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:26:43.765017 2721 raft_consensus.cc:504] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.766957 2721 leader_election.cc:288] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edf46ba8e80c406c989ce26488ae6b5a (127.14.127.194:45437), 471dc80bd2d3420aababd11ba5d59f33 (127.14.127.193:36053)
I20260923 20:26:43.773876 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.767532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:39953 (local address 127.14.127.193:36053)
0923 20:26:43.767728 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.767734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.767774 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:26:43.767857 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.767866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.767968 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.768073 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:43.768603 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.769598 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.771917 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.773044 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.773210 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.773286 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.773495 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.773648 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:26:43.773922 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.767222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.14.127.195:39953)
0923 20:26:43.767538 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:43.767564 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:43.767698 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:43.768117 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:43.768127 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:43.768154 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:43.768387 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.768399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.769754 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.769760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.771740 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.771769 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.773195 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.773199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.773334 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.773359 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:26:43.773431 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 20:26:43.773530 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2}
I20260923 20:26:43.774438 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.767401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.127.195:44181 (local address 127.14.127.194:45437)
0923 20:26:43.767630 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.767638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.768233 (+ 595us) server_negotiation.cc:407] Connection header received
0923 20:26:43.768416 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.768422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.768525 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.768613 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:26:43.769218 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.770072 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.772289 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.773485 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.773634 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.773694 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.774055 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.774219 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:26:43.775101 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.767359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.194:45437 (local address 127.14.127.195:44181)
0923 20:26:43.768141 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:43.768159 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:26:43.768274 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:26:43.768652 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:26:43.768657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:26:43.768685 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:26:43.769043 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.769055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.770241 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.770245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.772103 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:26:43.772116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.773662 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.773670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:26:43.773844 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.773883 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:26:43.774661 (+ 778us) client_negotiation.cc:240] Negotiation successful
0923 20:26:43.774802 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":481,"threads_started":1}
I20260923 20:26:43.775023 2390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb6071c11dd46a1b0faf2cb470efb77" candidate_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "471dc80bd2d3420aababd11ba5d59f33" is_pre_election: true
I20260923 20:26:43.775619 2390 raft_consensus.cc:2454] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e60c7c3c36e4304ac3e33f9aa323860 in term 0.
I20260923 20:26:43.776170 2521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb6071c11dd46a1b0faf2cb470efb77" candidate_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf46ba8e80c406c989ce26488ae6b5a" is_pre_election: true
I20260923 20:26:43.776296 2587 leader_election.cc:302] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 471dc80bd2d3420aababd11ba5d59f33, 8e60c7c3c36e4304ac3e33f9aa323860; no voters:
I20260923 20:26:43.776717 2721 raft_consensus.cc:2781] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:26:43.776734 2521 raft_consensus.cc:2454] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e60c7c3c36e4304ac3e33f9aa323860 in term 0.
I20260923 20:26:43.776885 2721 raft_consensus.cc:482] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:26:43.777004 2721 raft_consensus.cc:3037] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:43.779469 2721 raft_consensus.cc:504] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.780157 2721 leader_election.cc:288] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [CANDIDATE]: Term 1 election: Requested vote from peers edf46ba8e80c406c989ce26488ae6b5a (127.14.127.194:45437), 471dc80bd2d3420aababd11ba5d59f33 (127.14.127.193:36053)
I20260923 20:26:43.780632 2521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb6071c11dd46a1b0faf2cb470efb77" candidate_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf46ba8e80c406c989ce26488ae6b5a"
I20260923 20:26:43.780733 2390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb6071c11dd46a1b0faf2cb470efb77" candidate_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "471dc80bd2d3420aababd11ba5d59f33"
I20260923 20:26:43.780856 2521 raft_consensus.cc:3037] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:43.780997 2390 raft_consensus.cc:3037] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:43.784238 2521 raft_consensus.cc:2454] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e60c7c3c36e4304ac3e33f9aa323860 in term 1.
I20260923 20:26:43.784238 2390 raft_consensus.cc:2454] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e60c7c3c36e4304ac3e33f9aa323860 in term 1.
I20260923 20:26:43.784730 2584 leader_election.cc:302] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 471dc80bd2d3420aababd11ba5d59f33, 8e60c7c3c36e4304ac3e33f9aa323860, edf46ba8e80c406c989ce26488ae6b5a; no voters:
I20260923 20:26:43.785138 2721 raft_consensus.cc:2781] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:43.785673 2721 raft_consensus.cc:686] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 1 LEADER]: Becoming Leader. State: Replica: 8e60c7c3c36e4304ac3e33f9aa323860, State: Running, Role: LEADER
I20260923 20:26:43.786158 2721 consensus_queue.cc:237] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } }
I20260923 20:26:43.790889 2248 catalog_manager.cc:5495] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e60c7c3c36e4304ac3e33f9aa323860 (127.14.127.195). New cstate: current_term: 1 leader_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e60c7c3c36e4304ac3e33f9aa323860" member_type: VOTER last_known_addr { host: "127.14.127.195" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 } health_report { overall_health: UNKNOWN } } }
I20260923 20:26:43.807132 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.801968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.14.127.195:40435)
0923 20:26:43.802183 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.802187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.802204 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:26:43.802362 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.802366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.802441 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.802671 (+ 230us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:26:43.803048 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.803800 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.805080 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.805472 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.805560 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.805989 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:26:43.806492 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:26:43.806598 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.806763 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.806823 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:26:43.815227 2521 raft_consensus.cc:1261] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 1 FOLLOWER]: Refusing update from remote peer 8e60c7c3c36e4304ac3e33f9aa323860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:43.815227 2390 raft_consensus.cc:1261] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 1 FOLLOWER]: Refusing update from remote peer 8e60c7c3c36e4304ac3e33f9aa323860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:26:43.816351 2724 consensus_queue.cc:1035] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf46ba8e80c406c989ce26488ae6b5a" member_type: VOTER last_known_addr { host: "127.14.127.194" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:43.816744 2721 consensus_queue.cc:1035] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "471dc80bd2d3420aababd11ba5d59f33" member_type: VOTER last_known_addr { host: "127.14.127.193" port: 36053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:26:43.840548 2249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:43.840831 2249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:26:43.844424 2249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4da8203469004bbeaf583110a79ffd3e: Sending DeleteTablet for 3 replicas of tablet 9cb6071c11dd46a1b0faf2cb470efb77
I20260923 20:26:43.845636 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb6071c11dd46a1b0faf2cb470efb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:43 UTC) from {username='slave'} at 127.0.0.1:45942
I20260923 20:26:43.845770 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb6071c11dd46a1b0faf2cb470efb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:43 UTC) from {username='slave'} at 127.0.0.1:41018
I20260923 20:26:43.845983 2631 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cb6071c11dd46a1b0faf2cb470efb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:26:43 UTC) from {username='slave'} at 127.0.0.1:43246
I20260923 20:26:43.852746 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2308
I20260923 20:26:43.856947 2738 tablet_replica.cc:330] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: stopping tablet replica
I20260923 20:26:43.857717 2738 raft_consensus.cc:2229] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:26:43.858696 2738 raft_consensus.cc:2258] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:43.859719 2737 tablet_replica.cc:330] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: stopping tablet replica
I20260923 20:26:43.860324 2737 raft_consensus.cc:2229] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:43.860423 2736 tablet_replica.cc:330] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: stopping tablet replica
I20260923 20:26:43.860818 2737 raft_consensus.cc:2258] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:43.860917 2736 raft_consensus.cc:2229] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:26:43.861359 2736 raft_consensus.cc:2258] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:26:43.863559 2738 ts_tablet_manager.cc:1905] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:43.863550 2736 ts_tablet_manager.cc:1905] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:43.871311 2736 ts_tablet_manager.cc:1918] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:43.871611 2736 log.cc:1196] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-1/wal/wals/9cb6071c11dd46a1b0faf2cb470efb77
I20260923 20:26:43.871791 2737 ts_tablet_manager.cc:1905] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:26:43.872234 2736 ts_tablet_manager.cc:1939] T 9cb6071c11dd46a1b0faf2cb470efb77 P edf46ba8e80c406c989ce26488ae6b5a: Deleting consensus metadata
I20260923 20:26:43.874279 2738 ts_tablet_manager.cc:1918] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:43.874264 2233 catalog_manager.cc:4841] TS edf46ba8e80c406c989ce26488ae6b5a (127.14.127.194:45437): tablet 9cb6071c11dd46a1b0faf2cb470efb77 (table test-table [id=14d47b59ca024a9f935040c6811c5da4]) successfully deleted
I20260923 20:26:43.874460 2738 log.cc:1196] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-2/wal/wals/9cb6071c11dd46a1b0faf2cb470efb77
I20260923 20:26:43.875052 2738 ts_tablet_manager.cc:1939] T 9cb6071c11dd46a1b0faf2cb470efb77 P 8e60c7c3c36e4304ac3e33f9aa323860: Deleting consensus metadata
I20260923 20:26:43.876928 2236 catalog_manager.cc:4841] TS 8e60c7c3c36e4304ac3e33f9aa323860 (127.14.127.195:40435): tablet 9cb6071c11dd46a1b0faf2cb470efb77 (table test-table [id=14d47b59ca024a9f935040c6811c5da4]) successfully deleted
I20260923 20:26:43.878151 2737 ts_tablet_manager.cc:1918] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:26:43.878335 2737 log.cc:1196] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: Deleting WAL directory at /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194882707199-14847-0/minicluster-data/ts-0/wal/wals/9cb6071c11dd46a1b0faf2cb470efb77
I20260923 20:26:43.879005 2737 ts_tablet_manager.cc:1939] T 9cb6071c11dd46a1b0faf2cb470efb77 P 471dc80bd2d3420aababd11ba5d59f33: Deleting consensus metadata
I20260923 20:26:43.879101 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:43.854010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.14.127.193:36053)
0923 20:26:43.854227 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:43.854233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:43.866729 (+ 12496us) server_negotiation.cc:407] Connection header received
0923 20:26:43.867273 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:43.867280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:43.867388 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:43.867475 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:43.868277 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.869446 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.871450 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:43.872236 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:43.872336 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:43.877143 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:43.877159 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:43.877177 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:43.877991 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:43.878060 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:43.878087 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:43.878189 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:43.878475 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 20:26:43.878672 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:26:43.879878 2430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:43.983806 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 2308
I20260923 20:26:44.002100 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2439
W20260923 20:26:44.002146 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.127.193:36053 (error 108)
I20260923 20:26:44.004352 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.003773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41036)
0923 20:26:44.004034 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:44.004172 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260923 20:26:44.004729 2236 catalog_manager.cc:4584] TS 471dc80bd2d3420aababd11ba5d59f33 (127.14.127.193:36053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cb6071c11dd46a1b0faf2cb470efb77: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
I20260923 20:26:44.010545 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.003742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.14.127.194:45437)
0923 20:26:44.003928 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:44.003936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:44.003958 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:26:44.004024 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:44.004030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:44.004164 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:44.004349 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:44.004823 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:44.005664 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:44.007584 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:44.007967 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:44.008059 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:44.009194 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:44.009201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:44.009214 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:44.009784 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:44.009829 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:44.009848 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:44.009934 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:44.010096 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 20:26:44.010193 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:26:44.011238 2561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:44.058290 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.057820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41040)
0923 20:26:44.058031 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:44.058125 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:26:44.059437 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.059070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41044)
0923 20:26:44.059244 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:44.059311 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:26:44.120332 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.119679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41056)
0923 20:26:44.119969 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:44.120107 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 20:26:44.121625 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.121202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41060)
0923 20:26:44.121380 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:44.121471 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 20:26:44.147672 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 2439
I20260923 20:26:44.165921 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2569
I20260923 20:26:44.173954 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.166858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.14.127.195:40435)
0923 20:26:44.167077 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:44.167083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:44.167115 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:26:44.167190 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:44.167195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:44.167302 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:44.167419 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:44.167789 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:44.168987 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:44.170312 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:44.171036 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:44.171198 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:44.172212 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:44.172223 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:44.172242 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:44.173084 (+ 842us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:44.173181 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:44.173212 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:44.173328 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:44.173454 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 20:26:44.173558 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:26:44.174777 2691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:44.202282 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.201791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41072)
0923 20:26:44.202004 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:44.202136 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 20:26:44.203339 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.202952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41088)
0923 20:26:44.203160 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:44.203228 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 20:26:44.296507 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 2569
I20260923 20:26:44.316232 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2218
I20260923 20:26:44.317325 2279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:44.426517 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:44.425269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.193:36053 (local address 127.0.0.1:41100)
0923 20:26:44.426101 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0923 20:26:44.426239 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.193:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":670}
I20260923 20:26:44.427033 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 2218
2026-09-23T20:26:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2806 ms)
[----------] 6 tests from EncryptionPolicyTest (21276 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:44 dist-test-slave-wmd0 krb5kdc[2747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:44 dist-test-slave-wmd0 krb5kdc[2747](info): set up 2 sockets
Sep 23 20:26:44 dist-test-slave-wmd0 krb5kdc[2747](info): commencing operation
krb5kdc: starting...
W20260923 20:26:45.535215 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:45 dist-test-slave-wmd0 krb5kdc[2747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:26:45.790387 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:46.124078 2760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:46.124356 2760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:46.124488 2760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:46.131711 2760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:46.131844 2760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:46.131929 2760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:46.132040 2760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:46.132117 2760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:46.141947 2760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:46.143247 2760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:46.144584 2760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:46.156404 2765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:46.156335 2766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:46.156476 2768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:46.157272 2760 server_base.cc:1034] running on GCE node
I20260923 20:26:46.157984 2760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:46.159402 2760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:46.160640 2760 hybrid_clock.cc:648] HybridClock initialized: now 1790195206160573 us; error 51 us; skew 500 ppm
I20260923 20:26:46.161075 2760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:46 dist-test-slave-wmd0 krb5kdc[2747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:46.177409 2760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:26:46.179497 2760 webserver.cc:460] Webserver started at http://127.0.0.1:45139/ using document root <none> and password file <none>
I20260923 20:26:46.180284 2760 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:46.180435 2760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:46.180792 2760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:46.184671 2760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "d6744216bcd64857a84b628b9039b0d1"
format_stamp: "Formatted at 2026-09-23 20:26:46 on dist-test-slave-wmd0"
server_key: "7615253a016525ef20042c5d08da2c06"
server_key_iv: "fb659b8f968f5b9b0cf12b8134ef0d10"
server_key_version: "encryptionkey@0"
I20260923 20:26:46.185631 2760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "d6744216bcd64857a84b628b9039b0d1"
format_stamp: "Formatted at 2026-09-23 20:26:46 on dist-test-slave-wmd0"
server_key: "7615253a016525ef20042c5d08da2c06"
server_key_iv: "fb659b8f968f5b9b0cf12b8134ef0d10"
server_key_version: "encryptionkey@0"
I20260923 20:26:46.191957 2760 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260923 20:26:46.196853 2775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:46.198839 2760 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260923 20:26:46.199038 2760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "d6744216bcd64857a84b628b9039b0d1"
format_stamp: "Formatted at 2026-09-23 20:26:46 on dist-test-slave-wmd0"
server_key: "7615253a016525ef20042c5d08da2c06"
server_key_iv: "fb659b8f968f5b9b0cf12b8134ef0d10"
server_key_version: "encryptionkey@0"
I20260923 20:26:46.199226 2760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:46.215919 2760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:46.220013 2760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:46.220295 2760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:46.238534 2760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36061
I20260923 20:26:46.238554 2826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36061 every 8 connection(s)
I20260923 20:26:46.239959 2760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:46.243651 2827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:46.247300 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 2760
I20260923 20:26:46.247524 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:46.248339 14847 external_mini_cluster.cc:1428] Setting key 5c3f0f102b4f0fc50a2e067722f0062c
I20260923 20:26:46.258337 2827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1: Bootstrap starting.
I20260923 20:26:46.263096 2827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:46.264592 2827 log.cc:824] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:26:46 dist-test-slave-wmd0 krb5kdc[2747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:26:46.269258 2827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1: No bootstrap required, opened a new log
I20260923 20:26:46.279036 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:46.250299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49297 (local address 127.0.0.1:36061)
0923 20:26:46.251351 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:46.251363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:46.251416 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:26:46.253077 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:46.253100 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:46.253530 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:46.253872 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:46.255912 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:46.258189 (+ 2277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:46.260550 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:46.261839 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:46.262043 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:46.268704 (+ 6661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:46.268719 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:46.268746 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:46.268808 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:46.273678 (+ 4870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:46.274997 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:46.275006 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:46.275015 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:46.275262 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:46.275724 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:46.275729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:46.275732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:46.276504 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:46.276845 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:46.277212 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 20:26:46.277600 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":633,"threads_started":1}
I20260923 20:26:46.281584 2827 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744216bcd64857a84b628b9039b0d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36061 } }
I20260923 20:26:46.282071 2827 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:46.282186 2827 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6744216bcd64857a84b628b9039b0d1, State: Initialized, Role: FOLLOWER
I20260923 20:26:46.282840 2827 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744216bcd64857a84b628b9039b0d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36061 } }
I20260923 20:26:46.283109 2827 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:46.283263 2827 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:46.283463 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:26:46.283476 2827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:46.285640 2827 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744216bcd64857a84b628b9039b0d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36061 } }
I20260923 20:26:46.286208 2827 leader_election.cc:302] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6744216bcd64857a84b628b9039b0d1; no voters:
I20260923 20:26:46.287081 2827 leader_election.cc:288] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:46.287307 2834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:46.288952 2834 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [term 1 LEADER]: Becoming Leader. State: Replica: d6744216bcd64857a84b628b9039b0d1, State: Running, Role: LEADER
I20260923 20:26:46.289773 2827 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:46.289650 2834 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744216bcd64857a84b628b9039b0d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36061 } }
I20260923 20:26:46.297127 2837 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6744216bcd64857a84b628b9039b0d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744216bcd64857a84b628b9039b0d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36061 } } }
I20260923 20:26:46.297583 2837 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:46.296926 2838 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6744216bcd64857a84b628b9039b0d1. Latest consensus state: current_term: 1 leader_uuid: "d6744216bcd64857a84b628b9039b0d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6744216bcd64857a84b628b9039b0d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36061 } } }
I20260923 20:26:46.298204 2838 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:46.303809 2846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:46.309499 2846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:46.313474 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:46.289786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36913 (local address 127.0.0.1:36061)
0923 20:26:46.289967 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:46.289973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:46.290163 (+ 190us) server_negotiation.cc:407] Connection header received
0923 20:26:46.290482 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:46.290489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:46.290564 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:46.290673 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:46.293435 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:46.296068 (+ 2633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:46.300391 (+ 4323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:46.301319 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:46.301453 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:46.305649 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:46.305654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:46.305657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:46.305690 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:46.308841 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:46.309876 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:46.309885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:46.309891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:46.310076 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:46.310868 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:46.310877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:46.310883 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:46.311425 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:46.311758 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:46.312664 (+ 906us) server_negotiation.cc:299] Negotiation successful
0923 20:26:46.312932 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:26:46.323134 2846 catalog_manager.cc:1347] Generated new cluster ID: 46dbd8d4a647400abc905bbfb2619932
I20260923 20:26:46.323269 2846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:46.369832 2846 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:46.371198 2846 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:46.383433 2846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6744216bcd64857a84b628b9039b0d1: Generated new TSK 0
I20260923 20:26:46.384454 2846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:46.400478 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2760
I20260923 20:26:46.401939 2822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:46.497992 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 2760
W20260923 20:26:46.515763 2745 connection.cc:535] client connection to 127.0.0.1:36061 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36061): Connection reset by peer (error 104)
2026-09-23T20:26:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:46 dist-test-slave-wmd0 krb5kdc[2859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:46 dist-test-slave-wmd0 krb5kdc[2859](info): set up 2 sockets
Sep 23 20:26:46 dist-test-slave-wmd0 krb5kdc[2859](info): commencing operation
krb5kdc: starting...
W20260923 20:26:47.599233 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:47 dist-test-slave-wmd0 krb5kdc[2859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:26:47.857589 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:48.194432 2872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:48.194864 2872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:48.195039 2872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:48.202790 2872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:48.202932 2872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:48.203029 2872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:48.203125 2872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:48.203202 2872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:48.213351 2872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:48.214728 2872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:48.216077 2872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:48.228322 2877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:48.228322 2878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:48.228361 2880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:48.229355 2872 server_base.cc:1034] running on GCE node
I20260923 20:26:48.230083 2872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:48.231570 2872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:48.232788 2872 hybrid_clock.cc:648] HybridClock initialized: now 1790195208232742 us; error 48 us; skew 500 ppm
I20260923 20:26:48.233157 2872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:48 dist-test-slave-wmd0 krb5kdc[2859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:48.236470 2872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:26:48.237986 2872 webserver.cc:460] Webserver started at http://127.0.0.1:44941/ using document root <none> and password file <none>
I20260923 20:26:48.238683 2872 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:48.238795 2872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:48.239051 2872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:48.241910 2872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "3e730ec2e78b4d3fa15d5734dbc727f5"
format_stamp: "Formatted at 2026-09-23 20:26:48 on dist-test-slave-wmd0"
server_key: "9279eff40836896beae7e7a004fd9360"
server_key_iv: "e3fa71505294ac6ee8fa5c01f0f3c36a"
server_key_version: "encryptionkey@0"
I20260923 20:26:48.242686 2872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "3e730ec2e78b4d3fa15d5734dbc727f5"
format_stamp: "Formatted at 2026-09-23 20:26:48 on dist-test-slave-wmd0"
server_key: "9279eff40836896beae7e7a004fd9360"
server_key_iv: "e3fa71505294ac6ee8fa5c01f0f3c36a"
server_key_version: "encryptionkey@0"
I20260923 20:26:48.247848 2872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:26:48.251304 2887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:48.252768 2872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:26:48.252974 2872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "3e730ec2e78b4d3fa15d5734dbc727f5"
format_stamp: "Formatted at 2026-09-23 20:26:48 on dist-test-slave-wmd0"
server_key: "9279eff40836896beae7e7a004fd9360"
server_key_iv: "e3fa71505294ac6ee8fa5c01f0f3c36a"
server_key_version: "encryptionkey@0"
I20260923 20:26:48.253190 2872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:48.286316 2872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:48.290594 2872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:48.290916 2872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:48.309994 2872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44691
I20260923 20:26:48.310190 2938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44691 every 8 connection(s)
I20260923 20:26:48.311502 2872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:48.313242 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 2872
I20260923 20:26:48.313413 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:48.313954 14847 external_mini_cluster.cc:1428] Setting key b853c5de221ca341c0cdcd8a2ed7b94a
I20260923 20:26:48.316169 2939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:48.327461 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:48.315782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56487 (local address 127.0.0.1:44691)
0923 20:26:48.316606 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:48.316618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:48.316658 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:26:48.318263 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:48.318282 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:48.318747 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:48.319089 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:48.319699 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:48.321064 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:48.322915 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:48.323606 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:48.323742 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:48.325032 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:48.325039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:48.325053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:48.325633 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:48.325720 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:48.325747 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:48.325876 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:48.326188 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 20:26:48.326393 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":397,"threads_started":1}
I20260923 20:26:48.328892 2939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5: Bootstrap starting.
I20260923 20:26:48.331432 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:26:48.332242 2939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:48.333215 2939 log.cc:824] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:48.336630 2939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5: No bootstrap required, opened a new log
I20260923 20:26:48.342114 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:48.335789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54747 (local address 127.0.0.1:44691)
0923 20:26:48.335967 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:48.335976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:48.336394 (+ 418us) server_negotiation.cc:407] Connection header received
0923 20:26:48.336614 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:48.336621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:48.336701 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:48.336799 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:48.337302 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:48.338089 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:48.339535 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:48.340019 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:48.340171 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:48.341334 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:48.341339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:48.341343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:48.341360 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:48.341403 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:48.341418 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:48.341553 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:48.341753 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 20:26:48.341815 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:26:48.344676 2939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e730ec2e78b4d3fa15d5734dbc727f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44691 } }
I20260923 20:26:48.345054 2939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:48.345175 2939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e730ec2e78b4d3fa15d5734dbc727f5, State: Initialized, Role: FOLLOWER
I20260923 20:26:48.345743 2939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e730ec2e78b4d3fa15d5734dbc727f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44691 } }
I20260923 20:26:48.345975 2939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:48.346131 2939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:48.346309 2939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:48.348953 2939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e730ec2e78b4d3fa15d5734dbc727f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44691 } }
I20260923 20:26:48.349577 2939 leader_election.cc:302] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e730ec2e78b4d3fa15d5734dbc727f5; no voters:
I20260923 20:26:48.350142 2939 leader_election.cc:288] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:48.350291 2949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:48.350991 2949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [term 1 LEADER]: Becoming Leader. State: Replica: 3e730ec2e78b4d3fa15d5734dbc727f5, State: Running, Role: LEADER
I20260923 20:26:48.351579 2949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e730ec2e78b4d3fa15d5734dbc727f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44691 } }
I20260923 20:26:48.352273 2939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:48.355266 2951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e730ec2e78b4d3fa15d5734dbc727f5. Latest consensus state: current_term: 1 leader_uuid: "3e730ec2e78b4d3fa15d5734dbc727f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e730ec2e78b4d3fa15d5734dbc727f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44691 } } }
I20260923 20:26:48.355237 2950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e730ec2e78b4d3fa15d5734dbc727f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e730ec2e78b4d3fa15d5734dbc727f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44691 } } }
I20260923 20:26:48.355835 2950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:48.355832 2951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:48.357481 2958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:48.362840 2958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:48.371927 2958 catalog_manager.cc:1347] Generated new cluster ID: c5ebdfee0f8741308dd2ccd751e04504
I20260923 20:26:48.372092 2958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:48.392777 2958 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:48.394631 2958 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:48.405954 2958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e730ec2e78b4d3fa15d5734dbc727f5: Generated new TSK 0
I20260923 20:26:48.407107 2958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:48.418090 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2872
W20260923 20:26:48.419325 2934 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56487
W20260923 20:26:48.420244 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2872: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:26:48.420440 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 2872
W20260923 20:26:48.437933 2857 connection.cc:535] client connection to 127.0.0.1:44691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44691): Connection reset by peer (error 104)
2026-09-23T20:26:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:48 dist-test-slave-wmd0 krb5kdc[2970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:48 dist-test-slave-wmd0 krb5kdc[2970](info): set up 2 sockets
Sep 23 20:26:48 dist-test-slave-wmd0 krb5kdc[2970](info): commencing operation
krb5kdc: starting...
W20260923 20:26:49.519012 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:49 dist-test-slave-wmd0 krb5kdc[2970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:26:49.773665 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:50.109431 2983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:50.109741 2983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:50.109835 2983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:50.117069 2983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:50.117177 2983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:50.117223 2983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:50.117345 2983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:50.117424 2983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:50.127447 2983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:50.128834 2983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:50.130091 2983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:50.142151 2991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:50.142243 2989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:50.142462 2988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:50.144189 2983 server_base.cc:1034] running on GCE node
I20260923 20:26:50.144971 2983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:50.146476 2983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:50.147728 2983 hybrid_clock.cc:648] HybridClock initialized: now 1790195210147703 us; error 54 us; skew 500 ppm
I20260923 20:26:50.148101 2983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:50 dist-test-slave-wmd0 krb5kdc[2970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:50.151397 2983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:26:50.152909 2983 webserver.cc:460] Webserver started at http://127.0.0.1:33285/ using document root <none> and password file <none>
I20260923 20:26:50.153622 2983 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:50.153779 2983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:50.154098 2983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:50.157199 2983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "8cbee526c9d54fe59709d7da0fa17706"
format_stamp: "Formatted at 2026-09-23 20:26:50 on dist-test-slave-wmd0"
server_key: "836abb3c9ab74711867457bdf74564cb"
server_key_iv: "2913acc4c061f5c8c40b62e82790fe87"
server_key_version: "encryptionkey@0"
I20260923 20:26:50.158020 2983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "8cbee526c9d54fe59709d7da0fa17706"
format_stamp: "Formatted at 2026-09-23 20:26:50 on dist-test-slave-wmd0"
server_key: "836abb3c9ab74711867457bdf74564cb"
server_key_iv: "2913acc4c061f5c8c40b62e82790fe87"
server_key_version: "encryptionkey@0"
I20260923 20:26:50.163308 2983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 20:26:50.166929 2998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:50.168380 2983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:50.168599 2983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "8cbee526c9d54fe59709d7da0fa17706"
format_stamp: "Formatted at 2026-09-23 20:26:50 on dist-test-slave-wmd0"
server_key: "836abb3c9ab74711867457bdf74564cb"
server_key_iv: "2913acc4c061f5c8c40b62e82790fe87"
server_key_version: "encryptionkey@0"
I20260923 20:26:50.168825 2983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:50.174885 2983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:50.179111 2983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:50.179420 2983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:50.198161 2983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40149
I20260923 20:26:50.198210 3049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40149 every 8 connection(s)
I20260923 20:26:50.199658 2983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:50.203474 3050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:50.209159 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 2983
I20260923 20:26:50.209362 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:50.209800 14847 external_mini_cluster.cc:1428] Setting key a9409116b09d6d3bac5e7d97dd6f4ee1
I20260923 20:26:50.215054 3050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706: Bootstrap starting.
I20260923 20:26:50.219027 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:50.211801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50765 (local address 127.0.0.1:40149)
0923 20:26:50.212710 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:50.212725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:50.212773 (+ 48us) server_negotiation.cc:407] Connection header received
0923 20:26:50.215155 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:50.215182 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:50.215632 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:50.216048 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:50.217061 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:50.217070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:50.217087 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:50.217104 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:50.217186 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:50.217215 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:50.217463 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:50.217587 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 20:26:50.217765 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":387,"threads_started":1}
I20260923 20:26:50.219161 3050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:50.220242 3050 log.cc:824] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:50.223083 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:26:50.223603 3050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706: No bootstrap required, opened a new log
I20260923 20:26:50.230960 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:50.228344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46535 (local address 127.0.0.1:40149)
0923 20:26:50.228527 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:50.228534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:50.228929 (+ 395us) server_negotiation.cc:407] Connection header received
0923 20:26:50.229071 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:50.229076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:50.229140 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:50.229265 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:50.230134 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:50.230139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:50.230145 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:50.230169 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:50.230218 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:50.230243 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:50.230388 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:50.230547 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 20:26:50.230642 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:26:50.234015 3050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbee526c9d54fe59709d7da0fa17706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260923 20:26:50.234366 3050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:50.234440 3050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbee526c9d54fe59709d7da0fa17706, State: Initialized, Role: FOLLOWER
I20260923 20:26:50.235062 3050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbee526c9d54fe59709d7da0fa17706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260923 20:26:50.235275 3050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:50.235378 3050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:50.235518 3050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:50.238523 3050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbee526c9d54fe59709d7da0fa17706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260923 20:26:50.239058 3050 leader_election.cc:302] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cbee526c9d54fe59709d7da0fa17706; no voters:
I20260923 20:26:50.239595 3050 leader_election.cc:288] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:50.239755 3060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:50.240422 3060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [term 1 LEADER]: Becoming Leader. State: Replica: 8cbee526c9d54fe59709d7da0fa17706, State: Running, Role: LEADER
I20260923 20:26:50.240931 3060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbee526c9d54fe59709d7da0fa17706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } }
I20260923 20:26:50.241719 3050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:50.244431 3062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbee526c9d54fe59709d7da0fa17706. Latest consensus state: current_term: 1 leader_uuid: "8cbee526c9d54fe59709d7da0fa17706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbee526c9d54fe59709d7da0fa17706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } } }
I20260923 20:26:50.244828 3062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:50.245795 3061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cbee526c9d54fe59709d7da0fa17706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbee526c9d54fe59709d7da0fa17706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40149 } } }
I20260923 20:26:50.246100 3061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:50.247146 3068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:50.252980 3068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:50.262526 3068 catalog_manager.cc:1347] Generated new cluster ID: 0eb48d5d42114e47b3357aec033870ec
I20260923 20:26:50.262701 3068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:50.279197 3068 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:50.279639 3068 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:50.287470 3068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cbee526c9d54fe59709d7da0fa17706: Generated new TSK 0
I20260923 20:26:50.288497 3068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:50.294272 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2983
W20260923 20:26:50.295377 3045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50765
W20260923 20:26:50.296090 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 2983: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:26:50.296234 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 2983
2026-09-23T20:26:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:50 dist-test-slave-wmd0 krb5kdc[3081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:50 dist-test-slave-wmd0 krb5kdc[3081](info): set up 2 sockets
Sep 23 20:26:50 dist-test-slave-wmd0 krb5kdc[3081](info): commencing operation
krb5kdc: starting...
W20260923 20:26:51.407212 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:51 dist-test-slave-wmd0 krb5kdc[3081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:26:51.657856 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:51.995059 3094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:51.995374 3094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:51.995561 3094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:52.003027 3094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:52.003182 3094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:52.003293 3094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:52.003401 3094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:52.003482 3094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:52.014084 3094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:52.016201 3094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:52.018249 3094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:52.031150 3099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:52.031176 3102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:52.031106 3100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:52.032152 3094 server_base.cc:1034] running on GCE node
I20260923 20:26:52.033030 3094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:52.034575 3094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:52.035851 3094 hybrid_clock.cc:648] HybridClock initialized: now 1790195212035803 us; error 43 us; skew 500 ppm
I20260923 20:26:52.036242 3094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:52 dist-test-slave-wmd0 krb5kdc[3081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:52.043731 3094 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:26:52.045420 3094 webserver.cc:460] Webserver started at http://127.0.0.1:38565/ using document root <none> and password file <none>
I20260923 20:26:52.046149 3094 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:52.046306 3094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:52.046674 3094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:52.049758 3094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "5c3cd69f2b634f65b9fde75d4c8bae14"
format_stamp: "Formatted at 2026-09-23 20:26:52 on dist-test-slave-wmd0"
server_key: "cf4af3f029d75985bd6c65dc2c126b3d"
server_key_iv: "d202318f87345f1a7ef7fc9c7a89a173"
server_key_version: "encryptionkey@0"
I20260923 20:26:52.050567 3094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "5c3cd69f2b634f65b9fde75d4c8bae14"
format_stamp: "Formatted at 2026-09-23 20:26:52 on dist-test-slave-wmd0"
server_key: "cf4af3f029d75985bd6c65dc2c126b3d"
server_key_iv: "d202318f87345f1a7ef7fc9c7a89a173"
server_key_version: "encryptionkey@0"
I20260923 20:26:52.056031 3094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:26:52.059688 3109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:52.061075 3094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:26:52.061285 3094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "5c3cd69f2b634f65b9fde75d4c8bae14"
format_stamp: "Formatted at 2026-09-23 20:26:52 on dist-test-slave-wmd0"
server_key: "cf4af3f029d75985bd6c65dc2c126b3d"
server_key_iv: "d202318f87345f1a7ef7fc9c7a89a173"
server_key_version: "encryptionkey@0"
I20260923 20:26:52.061494 3094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:52.077550 3094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:52.081805 3094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:52.082123 3094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:52.101472 3094 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36051
I20260923 20:26:52.101503 3160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36051 every 8 connection(s)
I20260923 20:26:52.102916 3094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:52.103744 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3094
I20260923 20:26:52.103942 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:52.104478 14847 external_mini_cluster.cc:1428] Setting key e560d9da03fd73af97464ff606384117
I20260923 20:26:52.107730 3161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:26:52 dist-test-slave-wmd0 krb5kdc[3081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:26:52.121161 3161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14: Bootstrap starting.
I20260923 20:26:52.125717 3161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:52.127024 3161 log.cc:824] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:52.130705 3161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14: No bootstrap required, opened a new log
I20260923 20:26:52.130827 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:52.106420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39689 (local address 127.0.0.1:36051)
0923 20:26:52.107277 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:52.107295 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:52.107351 (+ 56us) server_negotiation.cc:407] Connection header received
0923 20:26:52.109661 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:52.109691 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:52.110208 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:52.110693 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:52.112099 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:52.114195 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:52.115583 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:52.116612 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:52.116803 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:52.121649 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:52.121662 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:52.121684 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:52.121742 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:52.125862 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:52.126701 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:52.126712 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:52.126723 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:52.126981 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:52.127432 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:52.127437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:52.127440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:52.128155 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:52.128386 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:52.128975 (+ 589us) server_negotiation.cc:299] Negotiation successful
0923 20:26:52.129441 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":385,"threads_started":1}
I20260923 20:26:52.134776 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:26:52.139658 3161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3cd69f2b634f65b9fde75d4c8bae14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } }
I20260923 20:26:52.140200 3161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:52.140313 3161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3cd69f2b634f65b9fde75d4c8bae14, State: Initialized, Role: FOLLOWER
I20260923 20:26:52.140937 3161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3cd69f2b634f65b9fde75d4c8bae14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } }
I20260923 20:26:52.141171 3161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:52.141310 3161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:52.141494 3161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:52.144387 3161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3cd69f2b634f65b9fde75d4c8bae14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } }
I20260923 20:26:52.145025 3161 leader_election.cc:302] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c3cd69f2b634f65b9fde75d4c8bae14; no voters:
I20260923 20:26:52.145697 3161 leader_election.cc:288] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:52.145807 3171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:52.147948 3161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:52.149121 3171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [term 1 LEADER]: Becoming Leader. State: Replica: 5c3cd69f2b634f65b9fde75d4c8bae14, State: Running, Role: LEADER
I20260923 20:26:52.149712 3171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3cd69f2b634f65b9fde75d4c8bae14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } }
I20260923 20:26:52.158810 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:52.139032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44255 (local address 127.0.0.1:36051)
0923 20:26:52.139211 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:52.139215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:52.139734 (+ 519us) server_negotiation.cc:407] Connection header received
0923 20:26:52.139865 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:52.139870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:52.139951 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:52.140062 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:52.141888 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:52.143076 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:52.144294 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:52.144939 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:52.145075 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:52.148867 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:52.148875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:52.148881 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:52.148923 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:52.153922 (+ 4999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:52.155135 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:52.155145 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:52.155151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:52.155353 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:52.156097 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:52.156104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:52.156107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:52.156504 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:52.156810 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:52.157939 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0923 20:26:52.158205 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:26:52.160120 3173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c3cd69f2b634f65b9fde75d4c8bae14. Latest consensus state: current_term: 1 leader_uuid: "5c3cd69f2b634f65b9fde75d4c8bae14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3cd69f2b634f65b9fde75d4c8bae14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } } }
I20260923 20:26:52.160509 3173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:52.161417 3172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c3cd69f2b634f65b9fde75d4c8bae14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3cd69f2b634f65b9fde75d4c8bae14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } } }
I20260923 20:26:52.161808 3172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14 [sys.catalog]: This master's current role is: LEADER
W20260923 20:26:52.166993 3185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:26:52.167133 3185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:26:52.167253 3186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:52.168886 3186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:52.177187 3186 catalog_manager.cc:1347] Generated new cluster ID: e2d3f358b8d747c8bfa4e08bbd6287f0
I20260923 20:26:52.177300 3186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:52.195137 3186 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:52.196425 3186 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:52.206175 3186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c3cd69f2b634f65b9fde75d4c8bae14: Generated new TSK 0
I20260923 20:26:52.207124 3186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:52.216499 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3094
I20260923 20:26:52.217597 3156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:52.310093 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3094
2026-09-23T20:26:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:52 dist-test-slave-wmd0 krb5kdc[3193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:52 dist-test-slave-wmd0 krb5kdc[3193](info): set up 2 sockets
Sep 23 20:26:52 dist-test-slave-wmd0 krb5kdc[3193](info): commencing operation
krb5kdc: starting...
W20260923 20:26:53.411203 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:53 dist-test-slave-wmd0 krb5kdc[3193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:26:53.665184 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:53.993379 3206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:53.993678 3206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:53.993772 3206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:54.000887 3206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:54.000993 3206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:54.001039 3206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:54.001099 3206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:54.001142 3206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:54.011175 3206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:54.012431 3206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:54.013775 3206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:54.028638 3211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:54.028673 3214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:54.028795 3212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:54.034956 3206 server_base.cc:1034] running on GCE node
I20260923 20:26:54.035790 3206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:54.037233 3206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:54.038463 3206 hybrid_clock.cc:648] HybridClock initialized: now 1790195214038423 us; error 59 us; skew 500 ppm
I20260923 20:26:54.038875 3206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:54 dist-test-slave-wmd0 krb5kdc[3193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:54.042040 3206 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:26:54.043601 3206 webserver.cc:460] Webserver started at http://127.0.0.1:35095/ using document root <none> and password file <none>
I20260923 20:26:54.044272 3206 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:54.044374 3206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:54.044632 3206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:54.047673 3206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "fd7d9d0ceac746c4957f096a139d4f45"
format_stamp: "Formatted at 2026-09-23 20:26:54 on dist-test-slave-wmd0"
server_key: "9a7082df074b262eed571e6f18ec24a0"
server_key_iv: "a41bddba076f32d81d6d946eb3b52f94"
server_key_version: "encryptionkey@0"
I20260923 20:26:54.048393 3206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "fd7d9d0ceac746c4957f096a139d4f45"
format_stamp: "Formatted at 2026-09-23 20:26:54 on dist-test-slave-wmd0"
server_key: "9a7082df074b262eed571e6f18ec24a0"
server_key_iv: "a41bddba076f32d81d6d946eb3b52f94"
server_key_version: "encryptionkey@0"
I20260923 20:26:54.053462 3206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:26:54.056895 3221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:54.058218 3206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 20:26:54.058408 3206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "fd7d9d0ceac746c4957f096a139d4f45"
format_stamp: "Formatted at 2026-09-23 20:26:54 on dist-test-slave-wmd0"
server_key: "9a7082df074b262eed571e6f18ec24a0"
server_key_iv: "a41bddba076f32d81d6d946eb3b52f94"
server_key_version: "encryptionkey@0"
I20260923 20:26:54.058602 3206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:54.079458 3206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:54.083815 3206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:54.084096 3206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:54.102682 3206 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44473
I20260923 20:26:54.102712 3272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44473 every 8 connection(s)
I20260923 20:26:54.104038 3206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:54.107631 3273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:54.110219 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3206
I20260923 20:26:54.110388 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:54.110939 14847 external_mini_cluster.cc:1428] Setting key b05aa8f52d610c04c77d344532c60e8a
I20260923 20:26:54.119719 3273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45: Bootstrap starting.
I20260923 20:26:54.123410 3273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:54.124202 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:54.112688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60451 (local address 127.0.0.1:44473)
0923 20:26:54.113527 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:54.113538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:54.113575 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:26:54.115302 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:54.115323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:54.115702 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:54.116021 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:54.116565 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:54.117880 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:54.119740 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:54.120440 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:54.120569 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:54.121808 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:54.121815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:54.121828 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:54.122354 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:54.122436 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:54.122465 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:54.122549 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:54.122820 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 20:26:54.123094 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":417,"threads_started":1}
I20260923 20:26:54.124608 3273 log.cc:824] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:54.127549 3273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45: No bootstrap required, opened a new log
I20260923 20:26:54.128034 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:26:54.138125 3273 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7d9d0ceac746c4957f096a139d4f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } }
I20260923 20:26:54.138614 3273 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:54.138758 3273 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7d9d0ceac746c4957f096a139d4f45, State: Initialized, Role: FOLLOWER
I20260923 20:26:54.139458 3273 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7d9d0ceac746c4957f096a139d4f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } }
I20260923 20:26:54.139745 3273 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:54.139794 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:54.133325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59831 (local address 127.0.0.1:44473)
0923 20:26:54.133488 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:54.133492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:54.133849 (+ 357us) server_negotiation.cc:407] Connection header received
0923 20:26:54.134002 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:54.134005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:54.134062 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:54.134150 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:54.134719 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:54.135415 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:54.137290 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:54.137707 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:54.137798 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:54.139004 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:54.139010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:54.139014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:54.139037 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:54.139082 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:54.139104 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:54.139203 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:54.139353 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 20:26:54.139442 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 20:26:54.139909 3273 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:54.140122 3273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:54.142477 3273 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7d9d0ceac746c4957f096a139d4f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } }
I20260923 20:26:54.142993 3273 leader_election.cc:302] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd7d9d0ceac746c4957f096a139d4f45; no voters:
I20260923 20:26:54.143522 3273 leader_election.cc:288] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:54.143671 3283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:54.144980 3283 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [term 1 LEADER]: Becoming Leader. State: Replica: fd7d9d0ceac746c4957f096a139d4f45, State: Running, Role: LEADER
I20260923 20:26:54.145507 3283 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7d9d0ceac746c4957f096a139d4f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } }
I20260923 20:26:54.145869 3273 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:54.148326 3284 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd7d9d0ceac746c4957f096a139d4f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7d9d0ceac746c4957f096a139d4f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } } }
I20260923 20:26:54.148305 3285 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7d9d0ceac746c4957f096a139d4f45. Latest consensus state: current_term: 1 leader_uuid: "fd7d9d0ceac746c4957f096a139d4f45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7d9d0ceac746c4957f096a139d4f45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44473 } } }
I20260923 20:26:54.148756 3284 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:54.148756 3285 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:54.161330 3289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:54.167829 3289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:54.176244 3289 catalog_manager.cc:1347] Generated new cluster ID: 42e9b0ca45a74ff58f46dd97034aa7c7
I20260923 20:26:54.176353 3289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:54.211961 3289 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:54.213416 3289 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:54.222101 3289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd7d9d0ceac746c4957f096a139d4f45: Generated new TSK 0
I20260923 20:26:54.222949 3289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:54.237051 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3206
W20260923 20:26:54.238098 3268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60451
W20260923 20:26:54.238793 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3206: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:26:54.238945 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3206
2026-09-23T20:26:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:54 dist-test-slave-wmd0 krb5kdc[3304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:54 dist-test-slave-wmd0 krb5kdc[3304](info): set up 2 sockets
Sep 23 20:26:54 dist-test-slave-wmd0 krb5kdc[3304](info): commencing operation
krb5kdc: starting...
W20260923 20:26:55.331573 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:55 dist-test-slave-wmd0 krb5kdc[3304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:26:55.582530 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:55.908160 3317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:55.908535 3317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:55.908718 3317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:55.916105 3317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:55.916260 3317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:55.916364 3317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:55.916473 3317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:55.916563 3317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:55.926595 3317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:55.927969 3317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:55.929252 3317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:55.940960 3325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:55.940907 3323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:55.940940 3322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:55.941664 3317 server_base.cc:1034] running on GCE node
I20260923 20:26:55.942445 3317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:55.943854 3317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:55.945135 3317 hybrid_clock.cc:648] HybridClock initialized: now 1790195215945063 us; error 44 us; skew 500 ppm
I20260923 20:26:55.945489 3317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:55 dist-test-slave-wmd0 krb5kdc[3304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:55.948580 3317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:26:55.949998 3317 webserver.cc:460] Webserver started at http://127.0.0.1:37145/ using document root <none> and password file <none>
I20260923 20:26:55.950764 3317 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:55.950915 3317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:55.951241 3317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:55.954211 3317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "37d504bea6d74097bddd09a01277e5df"
format_stamp: "Formatted at 2026-09-23 20:26:55 on dist-test-slave-wmd0"
server_key: "b65d65b39a3e8b4ea9d0e956a0c15723"
server_key_iv: "2883729cfc94f6ffe276c4dd76554104"
server_key_version: "encryptionkey@0"
I20260923 20:26:55.955039 3317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "37d504bea6d74097bddd09a01277e5df"
format_stamp: "Formatted at 2026-09-23 20:26:55 on dist-test-slave-wmd0"
server_key: "b65d65b39a3e8b4ea9d0e956a0c15723"
server_key_iv: "2883729cfc94f6ffe276c4dd76554104"
server_key_version: "encryptionkey@0"
I20260923 20:26:55.960323 3317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:26:55.963846 3332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:55.965195 3317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:26:55.965386 3317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "37d504bea6d74097bddd09a01277e5df"
format_stamp: "Formatted at 2026-09-23 20:26:55 on dist-test-slave-wmd0"
server_key: "b65d65b39a3e8b4ea9d0e956a0c15723"
server_key_iv: "2883729cfc94f6ffe276c4dd76554104"
server_key_version: "encryptionkey@0"
I20260923 20:26:55.965659 3317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:55.971563 3317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:55.975596 3317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:55.975911 3317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:55.993876 3317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36449
I20260923 20:26:55.993913 3383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36449 every 8 connection(s)
I20260923 20:26:55.995376 3317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:55.996423 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3317
I20260923 20:26:55.996603 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:55.997058 14847 external_mini_cluster.cc:1428] Setting key 9c774f99b014a16483fac37c8aeb7d09
I20260923 20:26:55.999488 3384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:56.006688 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:55.999137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40803 (local address 127.0.0.1:36449)
0923 20:26:56.000164 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:56.000181 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:56.000561 (+ 380us) server_negotiation.cc:407] Connection header received
0923 20:26:56.002695 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:56.002723 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:56.003209 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:56.003652 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:56.004655 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:56.004663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:56.004684 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:56.004714 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:56.004794 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:56.004820 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:56.005054 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:56.005214 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 20:26:56.005416 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":464,"threads_started":1}
I20260923 20:26:56.010378 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:26:56.014114 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df: Bootstrap starting.
I20260923 20:26:56.018721 3384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:56.020161 3384 log.cc:824] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:56.030462 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df: No bootstrap required, opened a new log
I20260923 20:26:56.031505 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:56.016046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40593 (local address 127.0.0.1:36449)
0923 20:26:56.026851 (+ 10805us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:56.026860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:56.027284 (+ 424us) server_negotiation.cc:407] Connection header received
0923 20:26:56.027521 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:56.027527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:56.027595 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:56.027690 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:56.030682 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:56.030688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:56.030693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:56.030714 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:56.030771 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:56.030797 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:56.030913 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:56.031142 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:26:56.031214 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10599}
I20260923 20:26:56.039290 3384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d504bea6d74097bddd09a01277e5df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } }
I20260923 20:26:56.039701 3384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:56.039826 3384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d504bea6d74097bddd09a01277e5df, State: Initialized, Role: FOLLOWER
I20260923 20:26:56.040472 3384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d504bea6d74097bddd09a01277e5df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } }
I20260923 20:26:56.040714 3384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:56.040869 3384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:56.041049 3384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:56.043599 3384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d504bea6d74097bddd09a01277e5df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } }
I20260923 20:26:56.044137 3384 leader_election.cc:302] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d504bea6d74097bddd09a01277e5df; no voters:
I20260923 20:26:56.044735 3384 leader_election.cc:288] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:56.044899 3394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:56.045933 3394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [term 1 LEADER]: Becoming Leader. State: Replica: 37d504bea6d74097bddd09a01277e5df, State: Running, Role: LEADER
I20260923 20:26:56.046550 3394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d504bea6d74097bddd09a01277e5df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } }
I20260923 20:26:56.047191 3384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:56.049721 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d504bea6d74097bddd09a01277e5df. Latest consensus state: current_term: 1 leader_uuid: "37d504bea6d74097bddd09a01277e5df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d504bea6d74097bddd09a01277e5df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } } }
I20260923 20:26:56.050189 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:56.049913 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d504bea6d74097bddd09a01277e5df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d504bea6d74097bddd09a01277e5df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36449 } } }
I20260923 20:26:56.050462 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:56.056030 3400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:56.062079 3400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:56.071444 3400 catalog_manager.cc:1347] Generated new cluster ID: a22e6fa7484146409bbf68dbf7683627
I20260923 20:26:56.071568 3400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:56.087846 3400 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:56.088294 3400 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:56.097126 3400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d504bea6d74097bddd09a01277e5df: Generated new TSK 0
I20260923 20:26:56.098155 3400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:56.109750 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3317
W20260923 20:26:56.111194 3379 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40803
W20260923 20:26:56.111876 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:26:56.112023 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3317
2026-09-23T20:26:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1872 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:56 dist-test-slave-wmd0 krb5kdc[3415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:56 dist-test-slave-wmd0 krb5kdc[3415](info): set up 2 sockets
Sep 23 20:26:56 dist-test-slave-wmd0 krb5kdc[3415](info): commencing operation
krb5kdc: starting...
W20260923 20:26:57.215183 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:57 dist-test-slave-wmd0 krb5kdc[3415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:26:57.469787 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:57.802893 3429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:57.803200 3429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:57.803392 3429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:57.811318 3429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:57.811476 3429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:57.811565 3429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:57.811666 3429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:57.811733 3429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:57.822474 3429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:57.823894 3429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:57.825238 3429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:57.837563 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:57.837597 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:57.837885 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:57.838317 3429 server_base.cc:1034] running on GCE node
I20260923 20:26:57.839133 3429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:57.840545 3429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:57.841791 3429 hybrid_clock.cc:648] HybridClock initialized: now 1790195217841747 us; error 65 us; skew 500 ppm
I20260923 20:26:57.842149 3429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:57 dist-test-slave-wmd0 krb5kdc[3415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:57.850922 3429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:26:57.852504 3429 webserver.cc:460] Webserver started at http://127.0.0.1:42393/ using document root <none> and password file <none>
I20260923 20:26:57.853240 3429 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:57.853394 3429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:57.853777 3429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:57.856901 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "a5a490ca30fe49789fdbd1c3edc35f63"
format_stamp: "Formatted at 2026-09-23 20:26:57 on dist-test-slave-wmd0"
server_key: "bc81e2f25b9da2b32ffe9cabefd74dcf"
server_key_iv: "5587d9f07499f5798a21874d9e0b0dfd"
server_key_version: "encryptionkey@0"
I20260923 20:26:57.857707 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "a5a490ca30fe49789fdbd1c3edc35f63"
format_stamp: "Formatted at 2026-09-23 20:26:57 on dist-test-slave-wmd0"
server_key: "bc81e2f25b9da2b32ffe9cabefd74dcf"
server_key_iv: "5587d9f07499f5798a21874d9e0b0dfd"
server_key_version: "encryptionkey@0"
I20260923 20:26:57.863042 3429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:26:57.866721 3444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:57.868103 3429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:26:57.868397 3429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "a5a490ca30fe49789fdbd1c3edc35f63"
format_stamp: "Formatted at 2026-09-23 20:26:57 on dist-test-slave-wmd0"
server_key: "bc81e2f25b9da2b32ffe9cabefd74dcf"
server_key_iv: "5587d9f07499f5798a21874d9e0b0dfd"
server_key_version: "encryptionkey@0"
I20260923 20:26:57.868633 3429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:57.897141 3429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:57.901629 3429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:57.901947 3429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:57.921271 3429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33223
I20260923 20:26:57.921291 3496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33223 every 8 connection(s)
I20260923 20:26:57.922746 3429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:57.925232 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3429
I20260923 20:26:57.925410 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:57.925855 14847 external_mini_cluster.cc:1428] Setting key 96abc8d871b7889905d4b681c5fd67e5
I20260923 20:26:57.926944 3497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:26:57 dist-test-slave-wmd0 krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:26:57.941618 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63: Bootstrap starting.
I20260923 20:26:57.945037 3497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:57.946159 3497 log.cc:824] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:57.949325 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63: No bootstrap required, opened a new log
I20260923 20:26:57.949355 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:57.927709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:51945 (local address 127.0.0.1:33223)
0923 20:26:57.928516 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:57.928527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:57.928568 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:26:57.930102 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:57.930124 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:57.930520 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:57.930943 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:57.932757 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:57.934088 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:57.935926 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:57.936585 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:57.936726 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:57.941545 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:57.941554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:57.941569 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:57.941619 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:57.944882 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:57.945817 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:57.945824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:57.945830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:57.946005 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:57.946564 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:57.946568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:57.946570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:57.947185 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:57.947614 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:57.948028 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 20:26:57.948355 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":393,"threads_started":1}
I20260923 20:26:57.953455 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:26:57.957643 3497 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a490ca30fe49789fdbd1c3edc35f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260923 20:26:57.958153 3497 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:57.958330 3497 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a490ca30fe49789fdbd1c3edc35f63, State: Initialized, Role: FOLLOWER
I20260923 20:26:57.959117 3497 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a490ca30fe49789fdbd1c3edc35f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260923 20:26:57.959427 3497 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:57.959581 3497 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:57.959777 3497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:57.962853 3497 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a490ca30fe49789fdbd1c3edc35f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260923 20:26:57.963457 3497 leader_election.cc:302] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5a490ca30fe49789fdbd1c3edc35f63; no voters:
I20260923 20:26:57.964148 3497 leader_election.cc:288] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:57.964294 3507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:57.965006 3507 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [term 1 LEADER]: Becoming Leader. State: Replica: a5a490ca30fe49789fdbd1c3edc35f63, State: Running, Role: LEADER
I20260923 20:26:57.965673 3507 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a490ca30fe49789fdbd1c3edc35f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260923 20:26:57.967222 3497 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:57.969033 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a490ca30fe49789fdbd1c3edc35f63. Latest consensus state: current_term: 1 leader_uuid: "a5a490ca30fe49789fdbd1c3edc35f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a490ca30fe49789fdbd1c3edc35f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } } }
I20260923 20:26:57.969646 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:57.970095 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5a490ca30fe49789fdbd1c3edc35f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a490ca30fe49789fdbd1c3edc35f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } } }
I20260923 20:26:57.971012 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:57.974915 3513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:57.977150 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:57.958025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:47527 (local address 127.0.0.1:33223)
0923 20:26:57.958262 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:57.958271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:57.958768 (+ 497us) server_negotiation.cc:407] Connection header received
0923 20:26:57.958934 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:57.958937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:57.959000 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:57.959082 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:57.961051 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:57.961846 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:57.963577 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:57.964233 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:57.964380 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:57.967731 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:57.967741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:57.967748 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:26:57.967796 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:57.973371 (+ 5575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:57.974494 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:57.974502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:57.974508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:57.974803 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:26:57.975314 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:26:57.975321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:26:57.975328 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:26:57.975794 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:57.976260 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:57.976600 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 20:26:57.976792 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:26:57.981998 3513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:57.992422 3513 catalog_manager.cc:1347] Generated new cluster ID: f0860283cf4042aeb3420daba983687f
I20260923 20:26:57.992556 3513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:58.016956 3513 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:58.019171 3513 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:58.026497 3513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5a490ca30fe49789fdbd1c3edc35f63: Generated new TSK 0
I20260923 20:26:58.027402 3513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:26:58.035006 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3429
I20260923 20:26:58.036931 3492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:26:58.128645 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3429
W20260923 20:26:58.146090 3413 connection.cc:535] client connection to 127.0.0.1:33223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33223): Connection reset by peer (error 104)
2026-09-23T20:26:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:26:58 dist-test-slave-wmd0 krb5kdc[3530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:26:58 dist-test-slave-wmd0 krb5kdc[3530](info): set up 2 sockets
Sep 23 20:26:58 dist-test-slave-wmd0 krb5kdc[3530](info): commencing operation
krb5kdc: starting...
W20260923 20:26:59.231289 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:26:59 dist-test-slave-wmd0 krb5kdc[3530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:26:59.480196 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:26:59.808272 3543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:26:59.808571 3543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:26:59.808746 3543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:26:59.815917 3543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:26:59.816025 3543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:26:59.816128 3543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:26:59.816215 3543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:26:59.816286 3543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:26:59.827652 3543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:26:59.828956 3543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:26:59.830272 3543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:26:59.841817 3549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:59.841853 3551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:26:59.841971 3548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:26:59.842557 3543 server_base.cc:1034] running on GCE node
I20260923 20:26:59.843364 3543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:26:59.844736 3543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:26:59.846004 3543 hybrid_clock.cc:648] HybridClock initialized: now 1790195219845967 us; error 48 us; skew 500 ppm
I20260923 20:26:59.846360 3543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:26:59 dist-test-slave-wmd0 krb5kdc[3530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:26:59.849505 3543 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:26:59.851205 3543 webserver.cc:460] Webserver started at http://127.0.0.1:37125/ using document root <none> and password file <none>
I20260923 20:26:59.851933 3543 fs_manager.cc:359] Metadata directory not provided
I20260923 20:26:59.852095 3543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:26:59.852404 3543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:26:59.855472 3543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "2c9eddbde4844e8897ef940f0f33b594"
format_stamp: "Formatted at 2026-09-23 20:26:59 on dist-test-slave-wmd0"
server_key: "9b6c77a5f37e801cb357bbb6180459d8"
server_key_iv: "e3a03ce12ada736866d246a6ab88caf4"
server_key_version: "encryptionkey@0"
I20260923 20:26:59.856252 3543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "2c9eddbde4844e8897ef940f0f33b594"
format_stamp: "Formatted at 2026-09-23 20:26:59 on dist-test-slave-wmd0"
server_key: "9b6c77a5f37e801cb357bbb6180459d8"
server_key_iv: "e3a03ce12ada736866d246a6ab88caf4"
server_key_version: "encryptionkey@0"
I20260923 20:26:59.861390 3543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:26:59.865103 3558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:26:59.866436 3543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:26:59.866688 3543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "2c9eddbde4844e8897ef940f0f33b594"
format_stamp: "Formatted at 2026-09-23 20:26:59 on dist-test-slave-wmd0"
server_key: "9b6c77a5f37e801cb357bbb6180459d8"
server_key_iv: "e3a03ce12ada736866d246a6ab88caf4"
server_key_version: "encryptionkey@0"
I20260923 20:26:59.866957 3543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:26:59.888595 3543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:26:59.892848 3543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:26:59.893163 3543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:26:59.912254 3543 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38005
I20260923 20:26:59.912290 3609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38005 every 8 connection(s)
I20260923 20:26:59.913655 3543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:26:59.914081 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3543
I20260923 20:26:59.914259 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:26:59.914969 14847 external_mini_cluster.cc:1428] Setting key b1465d8fd954aa36997d919c322e73f2
I20260923 20:26:59.918013 3610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:26:59.929637 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:59.917097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:43945 (local address 127.0.0.1:38005)
0923 20:26:59.918013 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:59.918030 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:59.918715 (+ 685us) server_negotiation.cc:407] Connection header received
0923 20:26:59.920467 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:59.920487 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:59.920871 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:59.921205 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:59.921767 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:59.923175 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:59.925078 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:59.925753 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:59.925942 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:59.927272 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:59.927279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:59.927295 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:59.927840 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:59.927921 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:59.927945 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:59.928078 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:59.928486 (+ 408us) server_negotiation.cc:299] Negotiation successful
0923 20:26:59.928692 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":402,"threads_started":1}
I20260923 20:26:59.930698 3610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594: Bootstrap starting.
I20260923 20:26:59.933547 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:26:59.934024 3610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594: Neither blocks nor log segments found. Creating new log.
I20260923 20:26:59.935096 3610 log.cc:824] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594: Log is configured to *not* fsync() on all Append() calls
I20260923 20:26:59.938253 3610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594: No bootstrap required, opened a new log
I20260923 20:26:59.944576 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:26:59.938048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:45411 (local address 127.0.0.1:38005)
0923 20:26:59.938211 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:26:59.938215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:26:59.938788 (+ 573us) server_negotiation.cc:407] Connection header received
0923 20:26:59.938990 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:26:59.938997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:26:59.939066 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:26:59.939169 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:26:59.939663 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:59.940470 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:59.941784 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:26:59.942503 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:26:59.942683 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:26:59.943804 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:26:59.943808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:26:59.943812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:26:59.943828 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:26:59.943865 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:26:59.943879 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:26:59.944033 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:26:59.944244 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 20:26:59.944300 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 20:26:59.946309 3610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9eddbde4844e8897ef940f0f33b594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260923 20:26:59.946681 3610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:26:59.946805 3610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9eddbde4844e8897ef940f0f33b594, State: Initialized, Role: FOLLOWER
I20260923 20:26:59.947434 3610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9eddbde4844e8897ef940f0f33b594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260923 20:26:59.947659 3610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:26:59.947806 3610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:26:59.947983 3610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:26:59.950248 3610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9eddbde4844e8897ef940f0f33b594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260923 20:26:59.950798 3610 leader_election.cc:302] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c9eddbde4844e8897ef940f0f33b594; no voters:
I20260923 20:26:59.951442 3610 leader_election.cc:288] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:26:59.951587 3620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:26:59.952322 3620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [term 1 LEADER]: Becoming Leader. State: Replica: 2c9eddbde4844e8897ef940f0f33b594, State: Running, Role: LEADER
I20260923 20:26:59.953003 3620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9eddbde4844e8897ef940f0f33b594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } }
I20260923 20:26:59.953740 3610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:26:59.955816 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c9eddbde4844e8897ef940f0f33b594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9eddbde4844e8897ef940f0f33b594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } } }
I20260923 20:26:59.956179 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:59.957552 3622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9eddbde4844e8897ef940f0f33b594. Latest consensus state: current_term: 1 leader_uuid: "2c9eddbde4844e8897ef940f0f33b594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9eddbde4844e8897ef940f0f33b594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38005 } } }
I20260923 20:26:59.957819 3622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594 [sys.catalog]: This master's current role is: LEADER
I20260923 20:26:59.957892 3626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:26:59.965583 3626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:26:59.974817 3626 catalog_manager.cc:1347] Generated new cluster ID: 06def865f60e49b38d514fa2fe204373
I20260923 20:26:59.974948 3626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:26:59.983937 3626 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:26:59.985183 3626 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:26:59.997460 3626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c9eddbde4844e8897ef940f0f33b594: Generated new TSK 0
I20260923 20:26:59.998306 3626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:00.016142 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3543
W20260923 20:27:00.017596 3605 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:43945
W20260923 20:27:00.018522 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3543: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:27:00.018754 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3543
W20260923 20:27:00.037849 3528 connection.cc:535] client connection to 127.0.0.1:38005 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38005): Connection reset by peer (error 104)
2026-09-23T20:27:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:00 dist-test-slave-wmd0 krb5kdc[3642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:00 dist-test-slave-wmd0 krb5kdc[3642](info): set up 2 sockets
Sep 23 20:27:00 dist-test-slave-wmd0 krb5kdc[3642](info): commencing operation
krb5kdc: starting...
W20260923 20:27:01.127243 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:01 dist-test-slave-wmd0 krb5kdc[3642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:27:01.383888 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:01.713907 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:01.714201 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:01.714294 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:01.721903 3655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:01.722026 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:01.722146 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:01.722260 3655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:01.722339 3655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:01.732894 3655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:01.734246 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:01.735733 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:01.748135 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:01.748137 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:01.748466 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:01.749105 3655 server_base.cc:1034] running on GCE node
I20260923 20:27:01.749960 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:01.751478 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:01.752714 3655 hybrid_clock.cc:648] HybridClock initialized: now 1790195221752664 us; error 33 us; skew 500 ppm
I20260923 20:27:01.753114 3655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:01 dist-test-slave-wmd0 krb5kdc[3642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:01.756369 3655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:27:01.757961 3655 webserver.cc:460] Webserver started at http://127.0.0.1:36187/ using document root <none> and password file <none>
I20260923 20:27:01.758725 3655 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:01.758859 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:01.759189 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:01.762156 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "ec268f4781164026bbc4d7d7c7e0d278"
format_stamp: "Formatted at 2026-09-23 20:27:01 on dist-test-slave-wmd0"
server_key: "2d35951d7ce470f66db45ac6e4bfdc39"
server_key_iv: "eab6bc13c96294ddb1b85acbd393064e"
server_key_version: "encryptionkey@0"
I20260923 20:27:01.762979 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "ec268f4781164026bbc4d7d7c7e0d278"
format_stamp: "Formatted at 2026-09-23 20:27:01 on dist-test-slave-wmd0"
server_key: "2d35951d7ce470f66db45ac6e4bfdc39"
server_key_iv: "eab6bc13c96294ddb1b85acbd393064e"
server_key_version: "encryptionkey@0"
I20260923 20:27:01.768152 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:27:01.771624 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:01.772889 3655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:27:01.773082 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "ec268f4781164026bbc4d7d7c7e0d278"
format_stamp: "Formatted at 2026-09-23 20:27:01 on dist-test-slave-wmd0"
server_key: "2d35951d7ce470f66db45ac6e4bfdc39"
server_key_iv: "eab6bc13c96294ddb1b85acbd393064e"
server_key_version: "encryptionkey@0"
I20260923 20:27:01.773294 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:01.779330 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:01.783180 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:01.783468 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:01.802049 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41845
I20260923 20:27:01.802090 3721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41845 every 8 connection(s)
I20260923 20:27:01.803583 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:01.807395 3722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:01.807858 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3655
I20260923 20:27:01.808036 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:01.808501 14847 external_mini_cluster.cc:1428] Setting key 071fbf3756ce5adc479e70ecce95f613
I20260923 20:27:01.816696 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:01.810257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:55049 (local address 127.0.0.1:41845)
0923 20:27:01.811138 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:01.811150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:01.811194 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:27:01.813328 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:01.813353 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:01.813716 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:01.814057 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:01.815162 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:01.815168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:01.815181 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:01.815195 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:01.815255 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:01.815273 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:01.815434 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:01.815576 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 20:27:01.815787 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":447,"threads_started":1}
I20260923 20:27:01.820446 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:01.820926 3722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278: Bootstrap starting.
I20260923 20:27:01.823923 3722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:01.825073 3722 log.cc:824] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:01.827440 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:01.824779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:50157 (local address 127.0.0.1:41845)
0923 20:27:01.824963 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:01.824969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:01.825469 (+ 500us) server_negotiation.cc:407] Connection header received
0923 20:27:01.825644 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:01.825650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:01.825715 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:01.825807 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:01.826828 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:01.826834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:01.826838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:01.826856 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:01.826915 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:01.826936 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:01.827051 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:01.827114 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 20:27:01.827166 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:27:01.828248 3722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278: No bootstrap required, opened a new log
I20260923 20:27:01.836843 3722 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec268f4781164026bbc4d7d7c7e0d278" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } }
I20260923 20:27:01.837219 3722 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:01.837325 3722 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec268f4781164026bbc4d7d7c7e0d278, State: Initialized, Role: FOLLOWER
I20260923 20:27:01.837949 3722 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec268f4781164026bbc4d7d7c7e0d278" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } }
I20260923 20:27:01.838192 3722 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:01.838341 3722 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:01.838517 3722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:01.840924 3722 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec268f4781164026bbc4d7d7c7e0d278" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } }
I20260923 20:27:01.841430 3722 leader_election.cc:302] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec268f4781164026bbc4d7d7c7e0d278; no voters:
I20260923 20:27:01.841982 3722 leader_election.cc:288] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:01.842090 3732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:01.842916 3732 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [term 1 LEADER]: Becoming Leader. State: Replica: ec268f4781164026bbc4d7d7c7e0d278, State: Running, Role: LEADER
I20260923 20:27:01.843447 3732 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec268f4781164026bbc4d7d7c7e0d278" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } }
I20260923 20:27:01.844135 3722 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:01.846362 3734 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec268f4781164026bbc4d7d7c7e0d278. Latest consensus state: current_term: 1 leader_uuid: "ec268f4781164026bbc4d7d7c7e0d278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec268f4781164026bbc4d7d7c7e0d278" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } } }
I20260923 20:27:01.846941 3734 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:01.846755 3733 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec268f4781164026bbc4d7d7c7e0d278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec268f4781164026bbc4d7d7c7e0d278" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41845 } } }
I20260923 20:27:01.847143 3733 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:01.851291 3740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:01.855901 3740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:01.863762 3740 catalog_manager.cc:1347] Generated new cluster ID: 1973840d81144c7aa7dda3d3cba74e52
I20260923 20:27:01.863867 3740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:01.879911 3740 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:01.880333 3740 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:01.891623 3740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec268f4781164026bbc4d7d7c7e0d278: Generated new TSK 0
I20260923 20:27:01.892601 3740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:01.908957 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3655
W20260923 20:27:01.910504 3717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:55049
W20260923 20:27:01.911284 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:27:01.911440 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3655
2026-09-23T20:27:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:02 dist-test-slave-wmd0 krb5kdc[3753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:02 dist-test-slave-wmd0 krb5kdc[3753](info): set up 2 sockets
Sep 23 20:27:02 dist-test-slave-wmd0 krb5kdc[3753](info): commencing operation
krb5kdc: starting...
W20260923 20:27:03.015139 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:03 dist-test-slave-wmd0 krb5kdc[3753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:27:03.261579 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:03.589103 3766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:03.589438 3766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:03.589605 3766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:03.596764 3766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:03.596918 3766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:03.597020 3766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:03.597134 3766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:03.597209 3766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:03.606940 3766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:03.608237 3766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:03.609525 3766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:03.621135 3772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:03.621096 3775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:03.621034 3773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:03.621973 3766 server_base.cc:1034] running on GCE node
I20260923 20:27:03.622799 3766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:03.624188 3766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:03.625423 3766 hybrid_clock.cc:648] HybridClock initialized: now 1790195223625380 us; error 58 us; skew 500 ppm
I20260923 20:27:03.625767 3766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:03 dist-test-slave-wmd0 krb5kdc[3753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:03.634478 3766 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:27:03.635973 3766 webserver.cc:460] Webserver started at http://127.0.0.1:46423/ using document root <none> and password file <none>
I20260923 20:27:03.636653 3766 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:03.636801 3766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:03.637112 3766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:03.640219 3766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "5c56b9f7924c46118bf21347695058e2"
format_stamp: "Formatted at 2026-09-23 20:27:03 on dist-test-slave-wmd0"
server_key: "87da63215108077cb03efb6ab1d46046"
server_key_iv: "1814b61c0b047ccae1c0de3160594231"
server_key_version: "encryptionkey@0"
I20260923 20:27:03.640972 3766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "5c56b9f7924c46118bf21347695058e2"
format_stamp: "Formatted at 2026-09-23 20:27:03 on dist-test-slave-wmd0"
server_key: "87da63215108077cb03efb6ab1d46046"
server_key_iv: "1814b61c0b047ccae1c0de3160594231"
server_key_version: "encryptionkey@0"
I20260923 20:27:03.645965 3766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 20:27:03.649353 3782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:03.650581 3766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:27:03.650862 3766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "5c56b9f7924c46118bf21347695058e2"
format_stamp: "Formatted at 2026-09-23 20:27:03 on dist-test-slave-wmd0"
server_key: "87da63215108077cb03efb6ab1d46046"
server_key_iv: "1814b61c0b047ccae1c0de3160594231"
server_key_version: "encryptionkey@0"
I20260923 20:27:03.651065 3766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:03.672268 3766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:03.676901 3766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:03.677215 3766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:03.696031 3766 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40255
I20260923 20:27:03.696079 3833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40255 every 8 connection(s)
I20260923 20:27:03.697392 3766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:03.701100 3834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:03.705163 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3766
I20260923 20:27:03.705346 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:03.705782 14847 external_mini_cluster.cc:1428] Setting key adf0490b7b222d569a14d1409bfe4a6c
I20260923 20:27:03.712152 3834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2: Bootstrap starting.
I20260923 20:27:03.715791 3834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:03.716945 3834 log.cc:824] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:03.720431 3834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2: No bootstrap required, opened a new log
Sep 23 20:27:03 dist-test-slave-wmd0 krb5kdc[3753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:27:03.728258 3834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c56b9f7924c46118bf21347695058e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260923 20:27:03.728755 3834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:03.728881 3834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c56b9f7924c46118bf21347695058e2, State: Initialized, Role: FOLLOWER
I20260923 20:27:03.729475 3834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c56b9f7924c46118bf21347695058e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260923 20:27:03.729707 3834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:03.729907 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:03.707564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:48605 (local address 127.0.0.1:40255)
0923 20:27:03.708310 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:03.708322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:03.708362 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:27:03.710541 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:03.710563 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:03.711021 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:03.711360 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:03.713211 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:03.714511 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:03.716285 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:03.716926 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:03.717054 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:03.721952 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:03.721962 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:03.721976 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:03.722013 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:03.725311 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:03.726148 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:03.726155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:03.726161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:03.726335 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:03.726913 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:03.726916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:03.726918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:03.727434 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:03.727784 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:03.728282 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 20:27:03.728593 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":331,"threads_started":1}
I20260923 20:27:03.729882 3834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:03.730151 3834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:03.732796 3834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c56b9f7924c46118bf21347695058e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260923 20:27:03.733325 3834 leader_election.cc:302] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c56b9f7924c46118bf21347695058e2; no voters:
I20260923 20:27:03.733759 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:03.734004 3834 leader_election.cc:288] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:03.734090 3839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:03.736585 3834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:03.739888 3839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c56b9f7924c46118bf21347695058e2, State: Running, Role: LEADER
I20260923 20:27:03.740715 3839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c56b9f7924c46118bf21347695058e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } }
I20260923 20:27:03.744493 3840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c56b9f7924c46118bf21347695058e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c56b9f7924c46118bf21347695058e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } } }
I20260923 20:27:03.744957 3840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:03.745584 3841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c56b9f7924c46118bf21347695058e2. Latest consensus state: current_term: 1 leader_uuid: "5c56b9f7924c46118bf21347695058e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c56b9f7924c46118bf21347695058e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40255 } } }
I20260923 20:27:03.747552 3841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:03.755007 3856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:03.762995 3856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:03.769785 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:03.742877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:36719 (local address 127.0.0.1:40255)
0923 20:27:03.743104 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:03.743112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:03.749720 (+ 6608us) server_negotiation.cc:407] Connection header received
0923 20:27:03.749968 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:03.749977 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:03.750090 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:03.750224 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:03.752881 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:03.754437 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:03.756974 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:03.757982 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:03.758125 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:03.761739 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:03.761745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:03.761750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:03.761790 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:03.765927 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:03.766990 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:03.766997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:03.767001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:03.767161 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:03.767772 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:03.767777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:03.767781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:03.768255 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:03.768527 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:03.769216 (+ 689us) server_negotiation.cc:299] Negotiation successful
0923 20:27:03.769406 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:27:03.775156 3856 catalog_manager.cc:1347] Generated new cluster ID: 4415165ffd4b4dc68214b1e941048685
I20260923 20:27:03.775274 3856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:03.786144 3856 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:03.787597 3856 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:03.797798 3856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c56b9f7924c46118bf21347695058e2: Generated new TSK 0
I20260923 20:27:03.798918 3856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:03.813350 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3766
I20260923 20:27:03.814411 3829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:03.906489 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3766
W20260923 20:27:03.923766 3751 connection.cc:535] client connection to 127.0.0.1:40255 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40255): Connection reset by peer (error 104)
2026-09-23T20:27:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:04 dist-test-slave-wmd0 krb5kdc[3867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:04 dist-test-slave-wmd0 krb5kdc[3867](info): set up 2 sockets
Sep 23 20:27:04 dist-test-slave-wmd0 krb5kdc[3867](info): commencing operation
krb5kdc: starting...
W20260923 20:27:05.011193 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:05 dist-test-slave-wmd0 krb5kdc[3867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:27:05.266371 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:05.596562 3880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:05.596876 3880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:05.597015 3880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:05.604115 3880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:05.604216 3880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:05.604261 3880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:05.604362 3880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:05.604447 3880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:05.614080 3880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40747
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:05.615336 3880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:05.616612 3880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:05.628165 3885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:05.628278 3886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:05.628304 3888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:05.629108 3880 server_base.cc:1034] running on GCE node
I20260923 20:27:05.629936 3880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:05.631321 3880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:05.632565 3880 hybrid_clock.cc:648] HybridClock initialized: now 1790195225632530 us; error 45 us; skew 500 ppm
I20260923 20:27:05.632915 3880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:05 dist-test-slave-wmd0 krb5kdc[3867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:05.635928 3880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:27:05.637357 3880 webserver.cc:460] Webserver started at http://127.0.0.1:37397/ using document root <none> and password file <none>
I20260923 20:27:05.638053 3880 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:05.638206 3880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:05.638512 3880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:05.641616 3880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "d73ae89531784c14afe64ee2ff72f29b"
format_stamp: "Formatted at 2026-09-23 20:27:05 on dist-test-slave-wmd0"
server_key: "88c377eee461f0c48dd7770d4a1d23a9"
server_key_iv: "8a24625f048dc32c7216d3122e1940f6"
server_key_version: "encryptionkey@0"
I20260923 20:27:05.642405 3880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "d73ae89531784c14afe64ee2ff72f29b"
format_stamp: "Formatted at 2026-09-23 20:27:05 on dist-test-slave-wmd0"
server_key: "88c377eee461f0c48dd7770d4a1d23a9"
server_key_iv: "8a24625f048dc32c7216d3122e1940f6"
server_key_version: "encryptionkey@0"
I20260923 20:27:05.647692 3880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260923 20:27:05.651216 3895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:05.652549 3880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:27:05.652729 3880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "d73ae89531784c14afe64ee2ff72f29b"
format_stamp: "Formatted at 2026-09-23 20:27:05 on dist-test-slave-wmd0"
server_key: "88c377eee461f0c48dd7770d4a1d23a9"
server_key_iv: "8a24625f048dc32c7216d3122e1940f6"
server_key_version: "encryptionkey@0"
I20260923 20:27:05.652920 3880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:05.670754 3880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:05.674598 3880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:05.674903 3880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:05.693409 3880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40747
I20260923 20:27:05.693450 3947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40747 every 8 connection(s)
I20260923 20:27:05.694885 3880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:05.698479 3948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:05.701313 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3880
I20260923 20:27:05.701506 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:05.701936 14847 external_mini_cluster.cc:1428] Setting key a2e95dc4ce4bdaeea7fd5d2760370983
I20260923 20:27:05.711655 3948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b: Bootstrap starting.
I20260923 20:27:05.715548 3948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:05.716888 3948 log.cc:824] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:05.717206 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:05.703854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:37785 (local address 127.0.0.1:40747)
0923 20:27:05.704614 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:05.704629 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:05.704678 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:27:05.707087 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:05.707116 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:05.707674 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:05.708126 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:05.708652 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:05.710689 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:05.712011 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:05.712995 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:05.713189 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:05.714375 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:05.714384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:05.714400 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:05.715134 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:05.715231 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:05.715263 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:05.715438 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:05.715719 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 20:27:05.715895 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":313,"threads_started":1}
I20260923 20:27:05.719939 3948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b: No bootstrap required, opened a new log
I20260923 20:27:05.720968 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:05.727692 3948 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73ae89531784c14afe64ee2ff72f29b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } }
I20260923 20:27:05.728083 3948 raft_consensus.cc:374] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:05.728219 3948 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73ae89531784c14afe64ee2ff72f29b, State: Initialized, Role: FOLLOWER
I20260923 20:27:05.728866 3948 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73ae89531784c14afe64ee2ff72f29b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } }
I20260923 20:27:05.729147 3948 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:05.729296 3948 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:05.729486 3948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:05.732126 3948 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73ae89531784c14afe64ee2ff72f29b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } }
I20260923 20:27:05.732693 3948 leader_election.cc:302] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73ae89531784c14afe64ee2ff72f29b; no voters:
I20260923 20:27:05.733311 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:05.725319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:59231 (local address 127.0.0.1:40747)
0923 20:27:05.725549 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:05.725558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:05.727214 (+ 1656us) server_negotiation.cc:407] Connection header received
0923 20:27:05.727357 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:05.727363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:05.727440 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:05.727548 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:05.728013 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:05.729149 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:05.730533 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:05.731244 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:05.731382 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:05.732493 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:05.732502 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:05.732507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:05.732527 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:05.732570 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:05.732590 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:05.732717 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:05.732875 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 20:27:05.732929 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:27:05.733426 3948 leader_election.cc:288] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:05.733577 3958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:05.735100 3958 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [term 1 LEADER]: Becoming Leader. State: Replica: d73ae89531784c14afe64ee2ff72f29b, State: Running, Role: LEADER
I20260923 20:27:05.735706 3948 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:05.735698 3958 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73ae89531784c14afe64ee2ff72f29b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } }
I20260923 20:27:05.742301 3960 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73ae89531784c14afe64ee2ff72f29b. Latest consensus state: current_term: 1 leader_uuid: "d73ae89531784c14afe64ee2ff72f29b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73ae89531784c14afe64ee2ff72f29b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } } }
I20260923 20:27:05.742338 3959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73ae89531784c14afe64ee2ff72f29b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73ae89531784c14afe64ee2ff72f29b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } } }
I20260923 20:27:05.742739 3959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:05.742736 3960 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b [sys.catalog]: This master's current role is: LEADER
W20260923 20:27:05.747715 3972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:27:05.747861 3972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:27:05.748027 3973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:05.749812 3973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:05.757691 3973 catalog_manager.cc:1347] Generated new cluster ID: ff926a31fa06463091cee328395ca28a
I20260923 20:27:05.757867 3973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:05.781450 3973 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:05.782944 3973 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:05.793705 3973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d73ae89531784c14afe64ee2ff72f29b: Generated new TSK 0
I20260923 20:27:05.794730 3973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:05.807888 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3880
W20260923 20:27:05.809083 3943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:37785
W20260923 20:27:05.809870 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3880: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:27:05.810016 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3880
W20260923 20:27:05.824254 3865 connection.cc:535] client connection to 127.0.0.1:40747 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40747): Connection reset by peer (error 104)
2026-09-23T20:27:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1904 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:05 dist-test-slave-wmd0 krb5kdc[3979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:05 dist-test-slave-wmd0 krb5kdc[3979](info): set up 2 sockets
Sep 23 20:27:05 dist-test-slave-wmd0 krb5kdc[3979](info): commencing operation
krb5kdc: starting...
W20260923 20:27:06.915271 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:07 dist-test-slave-wmd0 krb5kdc[3979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:27:07.165150 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:07.497689 3992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:07.498046 3992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:07.498214 3992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:07.505388 3992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:07.505548 3992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:07.505663 3992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:07.505774 3992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:07.505862 3992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:07.515697 3992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:07.517063 3992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:07.518464 3992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:07.531064 3997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:07.531103 4000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:07.531075 3998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:07.531939 3992 server_base.cc:1034] running on GCE node
I20260923 20:27:07.532696 3992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:07.534075 3992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:07.535280 3992 hybrid_clock.cc:648] HybridClock initialized: now 1790195227535248 us; error 31 us; skew 500 ppm
I20260923 20:27:07.535612 3992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:07 dist-test-slave-wmd0 krb5kdc[3979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:07.538757 3992 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:27:07.540210 3992 webserver.cc:460] Webserver started at http://127.0.0.1:38047/ using document root <none> and password file <none>
I20260923 20:27:07.540905 3992 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:07.541042 3992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:07.541368 3992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:07.544489 3992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "c84d70e6edc0473799bfecb20b303f01"
format_stamp: "Formatted at 2026-09-23 20:27:07 on dist-test-slave-wmd0"
server_key: "780e05f5ca151833b993d28fd371b87a"
server_key_iv: "6026c020e2012e4fcd1c8e2f4d920304"
server_key_version: "encryptionkey@0"
I20260923 20:27:07.545259 3992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "c84d70e6edc0473799bfecb20b303f01"
format_stamp: "Formatted at 2026-09-23 20:27:07 on dist-test-slave-wmd0"
server_key: "780e05f5ca151833b993d28fd371b87a"
server_key_iv: "6026c020e2012e4fcd1c8e2f4d920304"
server_key_version: "encryptionkey@0"
I20260923 20:27:07.550551 3992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:27:07.553932 4007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:07.555262 3992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:27:07.555444 3992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "c84d70e6edc0473799bfecb20b303f01"
format_stamp: "Formatted at 2026-09-23 20:27:07 on dist-test-slave-wmd0"
server_key: "780e05f5ca151833b993d28fd371b87a"
server_key_iv: "6026c020e2012e4fcd1c8e2f4d920304"
server_key_version: "encryptionkey@0"
I20260923 20:27:07.555648 3992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:07.561055 3992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:07.565310 3992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:07.565610 3992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:07.584414 3992 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44343
I20260923 20:27:07.584491 4058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44343 every 8 connection(s)
I20260923 20:27:07.585749 3992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:07.589130 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 3992
I20260923 20:27:07.589314 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:07.589572 4059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:07.589898 14847 external_mini_cluster.cc:1428] Setting key 52242fdfe03f321993b9f8a5f95b9250
I20260923 20:27:07.597399 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:07.591647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:42229 (local address 127.0.0.1:44343)
0923 20:27:07.592536 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:07.592547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:07.592592 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:27:07.594216 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:07.594235 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:07.594551 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:07.594927 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:07.595923 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:07.595929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:07.595941 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:07.595958 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:07.596017 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:07.596039 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:07.596209 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:07.596324 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 20:27:07.596524 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":450,"threads_started":1}
I20260923 20:27:07.600919 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:07.602751 4059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01: Bootstrap starting.
I20260923 20:27:07.606014 4059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:07.607170 4059 log.cc:824] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:07.607780 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:07.605325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:46129 (local address 127.0.0.1:44343)
0923 20:27:07.605464 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:07.605469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:07.605824 (+ 355us) server_negotiation.cc:407] Connection header received
0923 20:27:07.606003 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:07.606007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:07.606051 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:07.606178 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:07.607144 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:07.607149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:07.607154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:07.607182 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:07.607218 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:07.607232 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:07.607350 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:07.607460 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 20:27:07.607520 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 20:27:07.610085 4059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01: No bootstrap required, opened a new log
I20260923 20:27:07.617373 4059 raft_consensus.cc:348] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84d70e6edc0473799bfecb20b303f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } }
I20260923 20:27:07.617691 4059 raft_consensus.cc:374] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:07.617763 4059 raft_consensus.cc:729] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84d70e6edc0473799bfecb20b303f01, State: Initialized, Role: FOLLOWER
I20260923 20:27:07.618234 4059 consensus_queue.cc:260] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84d70e6edc0473799bfecb20b303f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } }
I20260923 20:27:07.618435 4059 raft_consensus.cc:388] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:07.618528 4059 raft_consensus.cc:482] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:07.618705 4059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:07.620849 4059 raft_consensus.cc:504] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84d70e6edc0473799bfecb20b303f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } }
I20260923 20:27:07.621423 4059 leader_election.cc:302] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c84d70e6edc0473799bfecb20b303f01; no voters:
I20260923 20:27:07.621999 4059 leader_election.cc:288] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:07.622148 4069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:07.622836 4069 raft_consensus.cc:686] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [term 1 LEADER]: Becoming Leader. State: Replica: c84d70e6edc0473799bfecb20b303f01, State: Running, Role: LEADER
I20260923 20:27:07.623364 4069 consensus_queue.cc:237] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84d70e6edc0473799bfecb20b303f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } }
I20260923 20:27:07.624137 4059 sys_catalog.cc:564] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:07.626015 4070 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c84d70e6edc0473799bfecb20b303f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84d70e6edc0473799bfecb20b303f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } } }
I20260923 20:27:07.626406 4070 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:07.628185 4074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:07.628152 4071 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84d70e6edc0473799bfecb20b303f01. Latest consensus state: current_term: 1 leader_uuid: "c84d70e6edc0473799bfecb20b303f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84d70e6edc0473799bfecb20b303f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44343 } } }
I20260923 20:27:07.628468 4071 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:07.633749 4074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:07.641970 4074 catalog_manager.cc:1347] Generated new cluster ID: 7426a28892f64f87a34a22d0ae18dda3
I20260923 20:27:07.642092 4074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:07.666644 4074 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:07.667083 4074 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:07.675122 4074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c84d70e6edc0473799bfecb20b303f01: Generated new TSK 0
I20260923 20:27:07.675909 4074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:07.690279 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3992
W20260923 20:27:07.691417 4054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:42229
W20260923 20:27:07.692137 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 3992: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:27:07.692294 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 3992
2026-09-23T20:27:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:07 dist-test-slave-wmd0 krb5kdc[4090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:07 dist-test-slave-wmd0 krb5kdc[4090](info): set up 2 sockets
Sep 23 20:27:07 dist-test-slave-wmd0 krb5kdc[4090](info): commencing operation
krb5kdc: starting...
W20260923 20:27:08.791217 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:08 dist-test-slave-wmd0 krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:27:09.045514 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:44485
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:44485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:09.374221 4103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:09.374491 4103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:09.374614 4103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:09.381688 4103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:09.381791 4103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:09.381837 4103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:09.381937 4103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:09.381999 4103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:09.391597 4103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:44485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:44485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:09.392890 4103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:09.394212 4103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:09.406423 4108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:09.406558 4111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:09.406596 4109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:09.407346 4103 server_base.cc:1034] running on GCE node
I20260923 20:27:09.408115 4103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:09.409531 4103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:09.410737 4103 hybrid_clock.cc:648] HybridClock initialized: now 1790195229410697 us; error 53 us; skew 500 ppm
I20260923 20:27:09.411128 4103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:09 dist-test-slave-wmd0 krb5kdc[4090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195229, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:09.421984 4103 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:27:09.423715 4103 webserver.cc:460] Webserver started at http://127.14.127.254:39995/ using document root <none> and password file <none>
I20260923 20:27:09.424460 4103 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:09.424621 4103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:09.424943 4103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:09.428330 4103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "e276dc2aaccb4cce99650245d6ec9d6c"
format_stamp: "Formatted at 2026-09-23 20:27:09 on dist-test-slave-wmd0"
server_key: "8bfa1681edaddbed2302664583ec68e4"
server_key_iv: "a70b776ab3f743bf482a5f641130d74a"
server_key_version: "encryptionkey@0"
I20260923 20:27:09.429150 4103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "e276dc2aaccb4cce99650245d6ec9d6c"
format_stamp: "Formatted at 2026-09-23 20:27:09 on dist-test-slave-wmd0"
server_key: "8bfa1681edaddbed2302664583ec68e4"
server_key_iv: "a70b776ab3f743bf482a5f641130d74a"
server_key_version: "encryptionkey@0"
I20260923 20:27:09.434489 4103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:27:09.437951 4118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:09.439428 4103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:27:09.439631 4103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "e276dc2aaccb4cce99650245d6ec9d6c"
format_stamp: "Formatted at 2026-09-23 20:27:09 on dist-test-slave-wmd0"
server_key: "8bfa1681edaddbed2302664583ec68e4"
server_key_iv: "a70b776ab3f743bf482a5f641130d74a"
server_key_version: "encryptionkey@0"
I20260923 20:27:09.439831 4103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:09.456792 4103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:09.460825 4103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:09.461127 4103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:09.479869 4103 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:44485
I20260923 20:27:09.479866 4169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:44485 every 8 connection(s)
I20260923 20:27:09.481277 4103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:09.485277 4170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:09.490166 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4103
I20260923 20:27:09.490372 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:09.490896 14847 external_mini_cluster.cc:1428] Setting key a1d03cabc787f1c709284c6fa9c642ce
I20260923 20:27:09.496284 4170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c: Bootstrap starting.
I20260923 20:27:09.499986 4170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:09.501219 4170 log.cc:824] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:09.504163 4170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c: No bootstrap required, opened a new log
Sep 23 20:27:09 dist-test-slave-wmd0 krb5kdc[4090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:27:09.514940 4170 raft_consensus.cc:348] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e276dc2aaccb4cce99650245d6ec9d6c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44485 } }
I20260923 20:27:09.515439 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:09.492623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36735 (local address 127.14.127.254:44485)
0923 20:27:09.493421 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:09.493432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:09.493478 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:27:09.495252 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:09.495272 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:09.495665 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:09.496004 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:09.497756 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:09.499161 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:09.500981 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:09.501800 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:09.502035 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:09.506095 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:09.506108 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:09.506128 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:09.506177 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:09.510328 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:09.511209 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:09.511220 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:09.511233 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:09.511458 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:09.511956 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:09.511961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:09.511965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:09.512772 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:09.513126 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:09.513598 (+ 472us) server_negotiation.cc:299] Negotiation successful
0923 20:27:09.513961 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":374,"threads_started":1}
I20260923 20:27:09.515465 4170 raft_consensus.cc:374] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:09.515626 4170 raft_consensus.cc:729] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e276dc2aaccb4cce99650245d6ec9d6c, State: Initialized, Role: FOLLOWER
I20260923 20:27:09.516350 4170 consensus_queue.cc:260] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e276dc2aaccb4cce99650245d6ec9d6c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44485 } }
I20260923 20:27:09.516623 4170 raft_consensus.cc:388] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:09.516776 4170 raft_consensus.cc:482] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:09.516992 4170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:09.519791 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:09.520125 4170 raft_consensus.cc:504] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e276dc2aaccb4cce99650245d6ec9d6c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44485 } }
I20260923 20:27:09.520680 4170 leader_election.cc:302] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e276dc2aaccb4cce99650245d6ec9d6c; no voters:
I20260923 20:27:09.521327 4170 leader_election.cc:288] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:09.521461 4175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:09.522883 4175 raft_consensus.cc:686] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [term 1 LEADER]: Becoming Leader. State: Replica: e276dc2aaccb4cce99650245d6ec9d6c, State: Running, Role: LEADER
I20260923 20:27:09.523464 4175 consensus_queue.cc:237] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e276dc2aaccb4cce99650245d6ec9d6c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44485 } }
I20260923 20:27:09.523833 4170 sys_catalog.cc:564] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:09.526252 4177 sys_catalog.cc:455] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e276dc2aaccb4cce99650245d6ec9d6c. Latest consensus state: current_term: 1 leader_uuid: "e276dc2aaccb4cce99650245d6ec9d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e276dc2aaccb4cce99650245d6ec9d6c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44485 } } }
I20260923 20:27:09.527190 4177 sys_catalog.cc:458] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:09.526859 4176 sys_catalog.cc:455] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e276dc2aaccb4cce99650245d6ec9d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e276dc2aaccb4cce99650245d6ec9d6c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 44485 } } }
I20260923 20:27:09.527623 4176 sys_catalog.cc:458] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:09.530061 4184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:09.539119 4184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:09.563771 4184 catalog_manager.cc:1347] Generated new cluster ID: fe0f880318224a6f81669a8d025bd9a8
I20260923 20:27:09.563890 4184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:09.566674 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:09.532740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57945 (local address 127.14.127.254:44485)
0923 20:27:09.532927 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:09.532937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:09.543366 (+ 10429us) server_negotiation.cc:407] Connection header received
0923 20:27:09.543574 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:09.543583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:09.543704 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:09.543864 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:09.546599 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:09.548246 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:09.550824 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:09.551821 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:09.551972 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:09.556465 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:09.556474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:09.556482 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:09.556539 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:09.561634 (+ 5095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:09.563054 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:09.563063 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:09.563070 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:09.563268 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:09.564031 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:09.564039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:09.564045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:09.564602 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:09.564946 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:09.565825 (+ 879us) server_negotiation.cc:299] Negotiation successful
0923 20:27:09.566091 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:27:09.593070 4184 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:09.595057 4184 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:09.607070 4184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e276dc2aaccb4cce99650245d6ec9d6c: Generated new TSK 0
I20260923 20:27:09.608111 4184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:09.616853 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4103
I20260923 20:27:09.618563 4165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:09.711510 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4103
W20260923 20:27:09.729336 4088 connection.cc:535] client connection to 127.14.127.254:44485 recv error: Network error: failed to read from TLS socket (remote: 127.14.127.254:44485): Connection reset by peer (error 104)
2026-09-23T20:27:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:09 dist-test-slave-wmd0 krb5kdc[4202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:09 dist-test-slave-wmd0 krb5kdc[4202](info): set up 2 sockets
Sep 23 20:27:09 dist-test-slave-wmd0 krb5kdc[4202](info): commencing operation
krb5kdc: starting...
W20260923 20:27:10.811208 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:10 dist-test-slave-wmd0 krb5kdc[4202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:27:11.081700 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:38799
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:11.409467 4215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:11.409737 4215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:11.409904 4215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:11.417095 4215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:11.417196 4215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:11.417241 4215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:11.417333 4215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:11.417397 4215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:11.427235 4215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:38799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:11.428476 4215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:11.429762 4215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:11.441581 4220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:11.441632 4223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:11.441682 4221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:11.444114 4215 server_base.cc:1034] running on GCE node
I20260923 20:27:11.444785 4215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:11.446066 4215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:11.447306 4215 hybrid_clock.cc:648] HybridClock initialized: now 1790195231447268 us; error 43 us; skew 500 ppm
I20260923 20:27:11.447677 4215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:11 dist-test-slave-wmd0 krb5kdc[4202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195231, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:11.450934 4215 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:27:11.452489 4215 webserver.cc:460] Webserver started at http://127.14.127.254:42555/ using document root <none> and password file <none>
I20260923 20:27:11.453202 4215 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:11.453364 4215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:11.453675 4215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:11.456748 4215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "06c26804701c4089b57476f18b349d4f"
format_stamp: "Formatted at 2026-09-23 20:27:11 on dist-test-slave-wmd0"
server_key: "353b9a35f038aceec5622bcb87c9ee27"
server_key_iv: "ee0f91c47c72c7e3a7c22b1be8446cb4"
server_key_version: "encryptionkey@0"
I20260923 20:27:11.457530 4215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "06c26804701c4089b57476f18b349d4f"
format_stamp: "Formatted at 2026-09-23 20:27:11 on dist-test-slave-wmd0"
server_key: "353b9a35f038aceec5622bcb87c9ee27"
server_key_iv: "ee0f91c47c72c7e3a7c22b1be8446cb4"
server_key_version: "encryptionkey@0"
I20260923 20:27:11.462749 4215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 20:27:11.466207 4230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:11.467729 4215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:27:11.467938 4215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "06c26804701c4089b57476f18b349d4f"
format_stamp: "Formatted at 2026-09-23 20:27:11 on dist-test-slave-wmd0"
server_key: "353b9a35f038aceec5622bcb87c9ee27"
server_key_iv: "ee0f91c47c72c7e3a7c22b1be8446cb4"
server_key_version: "encryptionkey@0"
I20260923 20:27:11.468137 4215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:11.486589 4215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:11.491048 4215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:11.491336 4215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:11.509641 4215 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:38799
I20260923 20:27:11.509662 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:38799 every 8 connection(s)
I20260923 20:27:11.511044 4215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:11.514693 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4215
I20260923 20:27:11.514801 4282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:11.514890 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:11.515306 14847 external_mini_cluster.cc:1428] Setting key 1f11b01fda1286c4ef4801e1ade3c40d
I20260923 20:27:11.528853 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:11.517048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44183 (local address 127.14.127.254:38799)
0923 20:27:11.517911 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:11.517922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:11.517965 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:27:11.519642 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:11.519664 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:11.520087 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:11.520412 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:11.521086 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:11.522436 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:11.524275 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:11.524935 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:11.525069 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:11.526411 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:11.526419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:11.526433 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:11.527121 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:11.527204 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:11.527236 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:11.527357 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:11.527687 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 20:27:11.527877 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":398,"threads_started":1}
I20260923 20:27:11.529034 4282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f: Bootstrap starting.
I20260923 20:27:11.532655 4282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:11.532959 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:11.533761 4282 log.cc:824] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:11.536791 4282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f: No bootstrap required, opened a new log
I20260923 20:27:11.543874 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:11.537562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52011 (local address 127.14.127.254:38799)
0923 20:27:11.537708 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:11.537712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:11.538245 (+ 533us) server_negotiation.cc:407] Connection header received
0923 20:27:11.538469 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:11.538473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:11.538537 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:11.538672 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:11.539158 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:11.539943 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:11.541275 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:11.541768 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:11.541911 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:11.543137 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:11.543142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:11.543145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:11.543162 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:11.543207 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:11.543222 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:11.543359 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:11.543550 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 20:27:11.543603 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:27:11.544883 4282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c26804701c4089b57476f18b349d4f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38799 } }
I20260923 20:27:11.545372 4282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:11.545490 4282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c26804701c4089b57476f18b349d4f, State: Initialized, Role: FOLLOWER
I20260923 20:27:11.546124 4282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c26804701c4089b57476f18b349d4f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38799 } }
I20260923 20:27:11.546356 4282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:11.546506 4282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:11.546759 4282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:11.548911 4282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c26804701c4089b57476f18b349d4f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38799 } }
I20260923 20:27:11.549471 4282 leader_election.cc:302] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c26804701c4089b57476f18b349d4f; no voters:
I20260923 20:27:11.550057 4282 leader_election.cc:288] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:11.550199 4292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:11.551565 4292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [term 1 LEADER]: Becoming Leader. State: Replica: 06c26804701c4089b57476f18b349d4f, State: Running, Role: LEADER
I20260923 20:27:11.552120 4292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c26804701c4089b57476f18b349d4f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38799 } }
I20260923 20:27:11.552409 4282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:11.554999 4294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c26804701c4089b57476f18b349d4f. Latest consensus state: current_term: 1 leader_uuid: "06c26804701c4089b57476f18b349d4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c26804701c4089b57476f18b349d4f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38799 } } }
I20260923 20:27:11.555466 4294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:11.555912 4293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c26804701c4089b57476f18b349d4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c26804701c4089b57476f18b349d4f" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 38799 } } }
I20260923 20:27:11.556253 4293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:11.557708 4299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:11.562932 4299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:11.572170 4299 catalog_manager.cc:1347] Generated new cluster ID: 799c1625a068452ca4e6c6bb85d2509d
I20260923 20:27:11.572288 4299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:11.590480 4299 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:11.591741 4299 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:11.600492 4299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c26804701c4089b57476f18b349d4f: Generated new TSK 0
I20260923 20:27:11.601266 4299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:11.611786 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4215
W20260923 20:27:11.612890 4277 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44183
W20260923 20:27:11.613644 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:27:11.613797 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4215
W20260923 20:27:11.631428 4200 connection.cc:535] client connection to 127.14.127.254:38799 recv error: Network error: failed to read from TLS socket (remote: 127.14.127.254:38799): Connection reset by peer (error 104)
2026-09-23T20:27:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:11 dist-test-slave-wmd0 krb5kdc[4313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:11 dist-test-slave-wmd0 krb5kdc[4313](info): set up 2 sockets
Sep 23 20:27:11 dist-test-slave-wmd0 krb5kdc[4313](info): commencing operation
krb5kdc: starting...
W20260923 20:27:12.711329 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:12 dist-test-slave-wmd0 krb5kdc[4313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:27:12.968228 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:34095
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:13.306542 4326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:13.306900 4326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:13.307072 4326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:13.314239 4326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:13.314338 4326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:13.314447 4326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:13.314567 4326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:13.314674 4326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:13.324401 4326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:34095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:13.325673 4326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:13.327031 4326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:13.338836 4334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:13.338864 4332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:13.339095 4331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:13.339819 4326 server_base.cc:1034] running on GCE node
I20260923 20:27:13.340590 4326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:13.342003 4326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:13.343233 4326 hybrid_clock.cc:648] HybridClock initialized: now 1790195233343184 us; error 46 us; skew 500 ppm
I20260923 20:27:13.343588 4326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:13 dist-test-slave-wmd0 krb5kdc[4313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:13.346652 4326 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:27:13.348117 4326 webserver.cc:460] Webserver started at http://127.14.127.254:46291/ using document root <none> and password file <none>
I20260923 20:27:13.348814 4326 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:13.348963 4326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:13.349277 4326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:13.352269 4326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "de84cd3bb19d4e35adb27f85c45ff94c"
format_stamp: "Formatted at 2026-09-23 20:27:13 on dist-test-slave-wmd0"
server_key: "119e5875a53e99dafb8222a17a1216f4"
server_key_iv: "5faa502eb579bff1c34a5c7a66d2d8b2"
server_key_version: "encryptionkey@0"
I20260923 20:27:13.353044 4326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "de84cd3bb19d4e35adb27f85c45ff94c"
format_stamp: "Formatted at 2026-09-23 20:27:13 on dist-test-slave-wmd0"
server_key: "119e5875a53e99dafb8222a17a1216f4"
server_key_iv: "5faa502eb579bff1c34a5c7a66d2d8b2"
server_key_version: "encryptionkey@0"
I20260923 20:27:13.358091 4326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:27:13.361717 4341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:13.363090 4326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:27:13.363273 4326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "de84cd3bb19d4e35adb27f85c45ff94c"
format_stamp: "Formatted at 2026-09-23 20:27:13 on dist-test-slave-wmd0"
server_key: "119e5875a53e99dafb8222a17a1216f4"
server_key_iv: "5faa502eb579bff1c34a5c7a66d2d8b2"
server_key_version: "encryptionkey@0"
I20260923 20:27:13.363477 4326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:13.369112 4326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:13.373103 4326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:13.373404 4326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:13.392071 4326 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:34095
I20260923 20:27:13.392112 4392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:34095 every 8 connection(s)
I20260923 20:27:13.393447 4326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:13.397128 4393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:13.402216 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4326
I20260923 20:27:13.402410 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:13.402895 14847 external_mini_cluster.cc:1428] Setting key 3bb4725f8f14b3f0d1a8088b50383cde
I20260923 20:27:13.407915 4393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c: Bootstrap starting.
I20260923 20:27:13.410773 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:13.404757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53879 (local address 127.14.127.254:34095)
0923 20:27:13.405585 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:13.405597 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:13.405641 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:27:13.407393 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:13.407417 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:13.407749 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:13.408134 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:13.409152 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:13.409160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:13.409171 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:13.409193 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:13.409246 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:13.409265 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:13.409433 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:13.409605 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 20:27:13.409810 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":389,"threads_started":1}
I20260923 20:27:13.411537 4393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:13.412631 4393 log.cc:824] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:13.414463 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:13.415664 4393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c: No bootstrap required, opened a new log
I20260923 20:27:13.421279 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:13.418787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41113 (local address 127.14.127.254:34095)
0923 20:27:13.419007 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:13.419016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:13.419434 (+ 418us) server_negotiation.cc:407] Connection header received
0923 20:27:13.419603 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:13.419607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:13.419653 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:13.419724 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:13.420627 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:13.420631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:13.420635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:13.420650 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:13.420688 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:13.420703 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:13.420793 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:13.420921 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 20:27:13.420968 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:27:13.424193 4393 raft_consensus.cc:348] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de84cd3bb19d4e35adb27f85c45ff94c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34095 } }
I20260923 20:27:13.424575 4393 raft_consensus.cc:374] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:13.424685 4393 raft_consensus.cc:729] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de84cd3bb19d4e35adb27f85c45ff94c, State: Initialized, Role: FOLLOWER
I20260923 20:27:13.425313 4393 consensus_queue.cc:260] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de84cd3bb19d4e35adb27f85c45ff94c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34095 } }
I20260923 20:27:13.425591 4393 raft_consensus.cc:388] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:13.425755 4393 raft_consensus.cc:482] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:13.425951 4393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:13.428210 4393 raft_consensus.cc:504] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de84cd3bb19d4e35adb27f85c45ff94c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34095 } }
I20260923 20:27:13.428731 4393 leader_election.cc:302] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de84cd3bb19d4e35adb27f85c45ff94c; no voters:
I20260923 20:27:13.429306 4393 leader_election.cc:288] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:13.429446 4403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:13.430884 4403 raft_consensus.cc:686] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [term 1 LEADER]: Becoming Leader. State: Replica: de84cd3bb19d4e35adb27f85c45ff94c, State: Running, Role: LEADER
I20260923 20:27:13.431459 4403 consensus_queue.cc:237] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de84cd3bb19d4e35adb27f85c45ff94c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34095 } }
I20260923 20:27:13.431581 4393 sys_catalog.cc:564] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:13.434599 4404 sys_catalog.cc:455] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de84cd3bb19d4e35adb27f85c45ff94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de84cd3bb19d4e35adb27f85c45ff94c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34095 } } }
I20260923 20:27:13.435081 4404 sys_catalog.cc:458] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:13.434499 4405 sys_catalog.cc:455] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [sys.catalog]: SysCatalogTable state changed. Reason: New leader de84cd3bb19d4e35adb27f85c45ff94c. Latest consensus state: current_term: 1 leader_uuid: "de84cd3bb19d4e35adb27f85c45ff94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de84cd3bb19d4e35adb27f85c45ff94c" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34095 } } }
I20260923 20:27:13.435762 4405 sys_catalog.cc:458] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:13.437021 4411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:13.442535 4411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:13.452399 4411 catalog_manager.cc:1347] Generated new cluster ID: 649fffeea83f4638b38a425fb1b3ecd6
I20260923 20:27:13.452540 4411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:13.466094 4411 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:13.466537 4411 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:13.480948 4411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de84cd3bb19d4e35adb27f85c45ff94c: Generated new TSK 0
I20260923 20:27:13.481734 4411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:13.485565 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4326
W20260923 20:27:13.486891 4388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53879
W20260923 20:27:13.487571 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4326: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:27:13.487692 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4326
2026-09-23T20:27:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:13 dist-test-slave-wmd0 krb5kdc[4424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:13 dist-test-slave-wmd0 krb5kdc[4424](info): set up 2 sockets
Sep 23 20:27:13 dist-test-slave-wmd0 krb5kdc[4424](info): commencing operation
krb5kdc: starting...
W20260923 20:27:14.595129 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:14 dist-test-slave-wmd0 krb5kdc[4424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:27:14.848151 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37059
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:37059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:15.175595 4437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:15.175945 4437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:15.176141 4437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:15.183214 4437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:15.183364 4437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:15.183473 4437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:15.183583 4437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:15.183655 4437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:15.193257 4437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:37059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:15.194531 4437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:15.195888 4437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:15.207646 4442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:15.207680 4443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:15.207732 4445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:15.208478 4437 server_base.cc:1034] running on GCE node
I20260923 20:27:15.209194 4437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:15.210503 4437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:15.211695 4437 hybrid_clock.cc:648] HybridClock initialized: now 1790195235211633 us; error 52 us; skew 500 ppm
I20260923 20:27:15.212000 4437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:15 dist-test-slave-wmd0 krb5kdc[4424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195235, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:15.218335 4437 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:27:15.219856 4437 webserver.cc:460] Webserver started at http://127.14.127.254:41065/ using document root <none> and password file <none>
I20260923 20:27:15.220566 4437 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:15.220707 4437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:15.221019 4437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:15.224018 4437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "d6c5e4745d234014abfa465636e56a9a"
format_stamp: "Formatted at 2026-09-23 20:27:15 on dist-test-slave-wmd0"
server_key: "34061d114e054653ca36b60ae03f86ed"
server_key_iv: "b4398419349f73ec92ffcef597d58192"
server_key_version: "encryptionkey@0"
I20260923 20:27:15.224803 4437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "d6c5e4745d234014abfa465636e56a9a"
format_stamp: "Formatted at 2026-09-23 20:27:15 on dist-test-slave-wmd0"
server_key: "34061d114e054653ca36b60ae03f86ed"
server_key_iv: "b4398419349f73ec92ffcef597d58192"
server_key_version: "encryptionkey@0"
I20260923 20:27:15.230181 4437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:27:15.233629 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:15.235006 4437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:27:15.235201 4437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "d6c5e4745d234014abfa465636e56a9a"
format_stamp: "Formatted at 2026-09-23 20:27:15 on dist-test-slave-wmd0"
server_key: "34061d114e054653ca36b60ae03f86ed"
server_key_iv: "b4398419349f73ec92ffcef597d58192"
server_key_version: "encryptionkey@0"
I20260923 20:27:15.235391 4437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:15.276367 4437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:15.280417 4437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:15.280715 4437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:15.299510 4437 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:37059
I20260923 20:27:15.299530 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:37059 every 8 connection(s)
I20260923 20:27:15.300968 4437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:15.302933 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4437
I20260923 20:27:15.303124 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:15.303582 14847 external_mini_cluster.cc:1428] Setting key 1e2c373b642f6c79e01c9c20ca15acc7
I20260923 20:27:15.305410 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:27:15 dist-test-slave-wmd0 krb5kdc[4424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:27:15.319514 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a: Bootstrap starting.
I20260923 20:27:15.323767 4505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:15.325088 4505 log.cc:824] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:15.328642 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a: No bootstrap required, opened a new log
I20260923 20:27:15.328881 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:15.305470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42693 (local address 127.14.127.254:37059)
0923 20:27:15.306359 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:15.306377 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:15.306429 (+ 52us) server_negotiation.cc:407] Connection header received
0923 20:27:15.308829 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:15.308859 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:15.309423 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:15.309911 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:15.311338 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:15.313383 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:15.314812 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:15.315888 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:15.316104 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:15.320206 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:15.320218 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:15.320236 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:15.320281 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:15.324301 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:15.325000 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:15.325009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:15.325018 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:15.325304 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:15.325707 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:15.325716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:15.325722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:15.326344 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:15.326683 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:15.327142 (+ 459us) server_negotiation.cc:299] Negotiation successful
0923 20:27:15.327571 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":396,"threads_started":1}
I20260923 20:27:15.332763 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:15.336906 4505 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5e4745d234014abfa465636e56a9a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37059 } }
I20260923 20:27:15.337389 4505 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:15.337523 4505 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c5e4745d234014abfa465636e56a9a, State: Initialized, Role: FOLLOWER
I20260923 20:27:15.338202 4505 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5e4745d234014abfa465636e56a9a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37059 } }
I20260923 20:27:15.338568 4505 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:15.338742 4505 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:15.338935 4505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:15.341931 4505 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5e4745d234014abfa465636e56a9a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37059 } }
I20260923 20:27:15.342528 4505 leader_election.cc:302] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6c5e4745d234014abfa465636e56a9a; no voters:
I20260923 20:27:15.343302 4505 leader_election.cc:288] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:15.343431 4515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:15.345999 4505 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:15.346897 4515 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [term 1 LEADER]: Becoming Leader. State: Replica: d6c5e4745d234014abfa465636e56a9a, State: Running, Role: LEADER
I20260923 20:27:15.347618 4515 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5e4745d234014abfa465636e56a9a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37059 } }
I20260923 20:27:15.350998 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6c5e4745d234014abfa465636e56a9a. Latest consensus state: current_term: 1 leader_uuid: "d6c5e4745d234014abfa465636e56a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5e4745d234014abfa465636e56a9a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37059 } } }
I20260923 20:27:15.351460 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:15.352141 4516 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6c5e4745d234014abfa465636e56a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6c5e4745d234014abfa465636e56a9a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37059 } } }
I20260923 20:27:15.352473 4516 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:15.354113 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:15.337157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45515 (local address 127.14.127.254:37059)
0923 20:27:15.337358 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:15.337366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:15.337946 (+ 580us) server_negotiation.cc:407] Connection header received
0923 20:27:15.338184 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:15.338192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:15.338252 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:15.338373 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:15.340267 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:15.341005 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:15.342723 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:15.343205 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:15.343375 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:15.346080 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:15.346098 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:15.346101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:15.346158 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:15.349459 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:15.350477 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:15.350482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:15.350485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:15.350691 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:15.351165 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:15.351171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:15.351174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:15.351596 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:15.351879 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:15.353226 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0923 20:27:15.353527 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:27:15.359549 4521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:15.364187 4521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:15.374214 4521 catalog_manager.cc:1347] Generated new cluster ID: 3d563e8c9acf4f23bbebd31e4490664f
I20260923 20:27:15.374344 4521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:15.387693 4521 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:15.389554 4521 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:15.397940 4521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6c5e4745d234014abfa465636e56a9a: Generated new TSK 0
I20260923 20:27:15.399003 4521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:15.405328 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4437
I20260923 20:27:15.406960 4500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:15.496865 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4437
2026-09-23T20:27:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:15 dist-test-slave-wmd0 krb5kdc[4538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:15 dist-test-slave-wmd0 krb5kdc[4538](info): set up 2 sockets
Sep 23 20:27:15 dist-test-slave-wmd0 krb5kdc[4538](info): commencing operation
krb5kdc: starting...
W20260923 20:27:16.603183 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:16 dist-test-slave-wmd0 krb5kdc[4538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:27:16.865931 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37739
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:17.195853 4551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:17.196175 4551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:17.196353 4551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:17.204483 4551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:17.204594 4551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:17.204686 4551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:17.204779 4551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:17.204845 4551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:17.214879 4551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:37739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:17.216207 4551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:17.217545 4551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:17.230064 4557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:17.230177 4556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:17.230177 4559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:17.231065 4551 server_base.cc:1034] running on GCE node
I20260923 20:27:17.231791 4551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:17.233212 4551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:17.234437 4551 hybrid_clock.cc:648] HybridClock initialized: now 1790195237234376 us; error 56 us; skew 500 ppm
I20260923 20:27:17.234809 4551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:17 dist-test-slave-wmd0 krb5kdc[4538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195237, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:17.237850 4551 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:27:17.239372 4551 webserver.cc:460] Webserver started at http://127.14.127.254:40459/ using document root <none> and password file <none>
I20260923 20:27:17.240051 4551 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:17.240211 4551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:17.240515 4551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:17.243522 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "96e16bd9401546e3b6309d0904c64167"
format_stamp: "Formatted at 2026-09-23 20:27:17 on dist-test-slave-wmd0"
server_key: "544336deaecf9bac6896888be142c408"
server_key_iv: "28a05a1ea58c8b8bb24eb6184537accc"
server_key_version: "encryptionkey@0"
I20260923 20:27:17.244272 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "96e16bd9401546e3b6309d0904c64167"
format_stamp: "Formatted at 2026-09-23 20:27:17 on dist-test-slave-wmd0"
server_key: "544336deaecf9bac6896888be142c408"
server_key_iv: "28a05a1ea58c8b8bb24eb6184537accc"
server_key_version: "encryptionkey@0"
I20260923 20:27:17.249269 4551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:27:17.252682 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:17.253975 4551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:27:17.254173 4551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "96e16bd9401546e3b6309d0904c64167"
format_stamp: "Formatted at 2026-09-23 20:27:17 on dist-test-slave-wmd0"
server_key: "544336deaecf9bac6896888be142c408"
server_key_iv: "28a05a1ea58c8b8bb24eb6184537accc"
server_key_version: "encryptionkey@0"
I20260923 20:27:17.254370 4551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:17.276458 4551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:17.280453 4551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:17.280750 4551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:17.299520 4551 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:37739
I20260923 20:27:17.299552 4617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:37739 every 8 connection(s)
I20260923 20:27:17.301133 4551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:17.304924 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:17.310938 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4551
I20260923 20:27:17.311086 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:17.311535 14847 external_mini_cluster.cc:1428] Setting key 7e691cf484e5b18642bca2a1cb68ee22
I20260923 20:27:17.316246 4618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167: Bootstrap starting.
I20260923 20:27:17.319386 4618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:17.320408 4618 log.cc:824] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:17.323788 4618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167: No bootstrap required, opened a new log
I20260923 20:27:17.325006 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:17.313422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58161 (local address 127.14.127.254:37739)
0923 20:27:17.314253 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:17.314265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:17.314308 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:27:17.315992 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:17.316015 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:17.316462 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:17.316854 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:17.317381 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:17.318743 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:17.320113 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:17.320840 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:17.321055 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:17.322298 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:17.322308 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:17.322325 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:17.323043 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:17.323150 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:17.323183 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:17.323304 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:17.323497 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 20:27:17.323746 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":385,"threads_started":1}
I20260923 20:27:17.328799 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:17.332553 4618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e16bd9401546e3b6309d0904c64167" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37739 } }
I20260923 20:27:17.333032 4618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:17.333149 4618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e16bd9401546e3b6309d0904c64167, State: Initialized, Role: FOLLOWER
I20260923 20:27:17.333714 4618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e16bd9401546e3b6309d0904c64167" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37739 } }
I20260923 20:27:17.333941 4618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:17.334096 4618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:17.334267 4618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:17.337697 4618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e16bd9401546e3b6309d0904c64167" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37739 } }
I20260923 20:27:17.338245 4618 leader_election.cc:302] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96e16bd9401546e3b6309d0904c64167; no voters:
I20260923 20:27:17.338913 4618 leader_election.cc:288] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:17.339052 4628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:17.340929 4618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:17.341627 4628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [term 1 LEADER]: Becoming Leader. State: Replica: 96e16bd9401546e3b6309d0904c64167, State: Running, Role: LEADER
I20260923 20:27:17.342265 4628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e16bd9401546e3b6309d0904c64167" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37739 } }
I20260923 20:27:17.343307 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:17.333062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59023 (local address 127.14.127.254:37739)
0923 20:27:17.333299 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:17.333308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:17.335243 (+ 1935us) server_negotiation.cc:407] Connection header received
0923 20:27:17.335556 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:17.335562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:17.335665 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:17.335789 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:17.336557 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:17.337615 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:17.339298 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:17.340210 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:17.340345 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:17.342011 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:17.342021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:17.342027 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:17.342053 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:17.342108 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:17.342134 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:17.342252 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:17.342706 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 20:27:17.342829 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:27:17.351912 4629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96e16bd9401546e3b6309d0904c64167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e16bd9401546e3b6309d0904c64167" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37739 } } }
I20260923 20:27:17.352010 4630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e16bd9401546e3b6309d0904c64167. Latest consensus state: current_term: 1 leader_uuid: "96e16bd9401546e3b6309d0904c64167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e16bd9401546e3b6309d0904c64167" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 37739 } } }
I20260923 20:27:17.352332 4630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:17.352332 4629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:17.355473 4638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:17.360540 4638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:17.368712 4638 catalog_manager.cc:1347] Generated new cluster ID: d70ca0c409df4840bbe65b290c46a5ee
I20260923 20:27:17.368842 4638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:17.388551 4638 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:17.390404 4638 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:17.399646 4638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96e16bd9401546e3b6309d0904c64167: Generated new TSK 0
I20260923 20:27:17.400664 4638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:17.407985 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4551
W20260923 20:27:17.408998 4613 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58161
W20260923 20:27:17.409660 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4551: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:27:17.409818 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4551
2026-09-23T20:27:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:17 dist-test-slave-wmd0 krb5kdc[4649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:17 dist-test-slave-wmd0 krb5kdc[4649](info): set up 2 sockets
Sep 23 20:27:17 dist-test-slave-wmd0 krb5kdc[4649](info): commencing operation
krb5kdc: starting...
W20260923 20:27:18.515302 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:18 dist-test-slave-wmd0 krb5kdc[4649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:27:18.771709 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:45477
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:45477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:19.104401 4662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:19.104746 4662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:19.104923 4662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:19.112735 4662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:19.112885 4662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:19.112967 4662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:19.113065 4662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:19.113131 4662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:19.123148 4662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:45477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:19.124498 4662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:19.125854 4662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:19.138096 4670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:19.138147 4667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:19.138046 4668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:19.138732 4662 server_base.cc:1034] running on GCE node
I20260923 20:27:19.139552 4662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:19.140973 4662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:19.142218 4662 hybrid_clock.cc:648] HybridClock initialized: now 1790195239142183 us; error 46 us; skew 500 ppm
I20260923 20:27:19.142585 4662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:19 dist-test-slave-wmd0 krb5kdc[4649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195239, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:19.145655 4662 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:27:19.147265 4662 webserver.cc:460] Webserver started at http://127.14.127.254:33845/ using document root <none> and password file <none>
I20260923 20:27:19.147955 4662 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:19.148108 4662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:19.148416 4662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:19.151454 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "c46f286df0dd4bf98d13993188f09ce0"
format_stamp: "Formatted at 2026-09-23 20:27:19 on dist-test-slave-wmd0"
server_key: "a926e3de6b431956cd764f8ef7f51b96"
server_key_iv: "b887c335eb3f15787da69751de0b441d"
server_key_version: "encryptionkey@0"
I20260923 20:27:19.152235 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "c46f286df0dd4bf98d13993188f09ce0"
format_stamp: "Formatted at 2026-09-23 20:27:19 on dist-test-slave-wmd0"
server_key: "a926e3de6b431956cd764f8ef7f51b96"
server_key_iv: "b887c335eb3f15787da69751de0b441d"
server_key_version: "encryptionkey@0"
I20260923 20:27:19.157312 4662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:27:19.160738 4677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:19.162042 4662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:27:19.162230 4662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "c46f286df0dd4bf98d13993188f09ce0"
format_stamp: "Formatted at 2026-09-23 20:27:19 on dist-test-slave-wmd0"
server_key: "a926e3de6b431956cd764f8ef7f51b96"
server_key_iv: "b887c335eb3f15787da69751de0b441d"
server_key_version: "encryptionkey@0"
I20260923 20:27:19.162432 4662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:19.168455 4662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:19.172379 4662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:19.172685 4662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:19.191695 4662 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:45477
I20260923 20:27:19.191730 4728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:45477 every 8 connection(s)
I20260923 20:27:19.193125 4662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:19.196844 4729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:19.197571 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4662
I20260923 20:27:19.197760 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:19.198239 14847 external_mini_cluster.cc:1428] Setting key 830cc9f44169337ce75c65a4dddf31bc
I20260923 20:27:19.207085 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:19.200277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51905 (local address 127.14.127.254:45477)
0923 20:27:19.201243 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:19.201259 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:19.201641 (+ 382us) server_negotiation.cc:407] Connection header received
0923 20:27:19.203406 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:19.203433 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:19.203879 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:19.204323 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:19.205215 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:19.205222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:19.205237 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:19.205253 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:19.205315 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:19.205344 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:19.205553 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:19.205686 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 20:27:19.205863 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":419,"threads_started":1}
I20260923 20:27:19.210780 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:19.211165 4729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0: Bootstrap starting.
I20260923 20:27:19.213946 4729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:19.215163 4729 log.cc:824] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:19.217648 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:19.214990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39361 (local address 127.14.127.254:45477)
0923 20:27:19.215177 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:19.215186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:19.215725 (+ 539us) server_negotiation.cc:407] Connection header received
0923 20:27:19.215920 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:19.215925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:19.215985 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:19.216096 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:19.217027 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:19.217032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:19.217037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:27:19.217054 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:19.217100 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:27:19.217115 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:19.217234 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:19.217324 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 20:27:19.217383 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:27:19.218159 4729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0: No bootstrap required, opened a new log
I20260923 20:27:19.225754 4729 raft_consensus.cc:348] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46f286df0dd4bf98d13993188f09ce0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 45477 } }
I20260923 20:27:19.226130 4729 raft_consensus.cc:374] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:27:19.226238 4729 raft_consensus.cc:729] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46f286df0dd4bf98d13993188f09ce0, State: Initialized, Role: FOLLOWER
I20260923 20:27:19.226928 4729 consensus_queue.cc:260] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46f286df0dd4bf98d13993188f09ce0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 45477 } }
I20260923 20:27:19.227195 4729 raft_consensus.cc:388] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:27:19.227352 4729 raft_consensus.cc:482] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:27:19.227530 4729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:19.229612 4729 raft_consensus.cc:504] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46f286df0dd4bf98d13993188f09ce0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 45477 } }
I20260923 20:27:19.230100 4729 leader_election.cc:302] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c46f286df0dd4bf98d13993188f09ce0; no voters:
I20260923 20:27:19.230762 4729 leader_election.cc:288] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:27:19.230907 4739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:19.231889 4739 raft_consensus.cc:686] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [term 1 LEADER]: Becoming Leader. State: Replica: c46f286df0dd4bf98d13993188f09ce0, State: Running, Role: LEADER
I20260923 20:27:19.232455 4739 consensus_queue.cc:237] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46f286df0dd4bf98d13993188f09ce0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 45477 } }
I20260923 20:27:19.232882 4729 sys_catalog.cc:564] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:19.235312 4740 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c46f286df0dd4bf98d13993188f09ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46f286df0dd4bf98d13993188f09ce0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 45477 } } }
I20260923 20:27:19.235538 4741 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46f286df0dd4bf98d13993188f09ce0. Latest consensus state: current_term: 1 leader_uuid: "c46f286df0dd4bf98d13993188f09ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46f286df0dd4bf98d13993188f09ce0" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 45477 } } }
I20260923 20:27:19.235831 4740 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:19.235832 4741 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:19.237967 4748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:19.243113 4748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:19.252132 4748 catalog_manager.cc:1347] Generated new cluster ID: 15655f494ee24c008151ea73b864abb6
I20260923 20:27:19.252270 4748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:19.264842 4748 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:19.265354 4748 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:19.278080 4748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c46f286df0dd4bf98d13993188f09ce0: Generated new TSK 0
I20260923 20:27:19.279016 4748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:19.295626 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4662
W20260923 20:27:19.296736 4724 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51905
W20260923 20:27:19.297462 14847 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4662: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:27:19.297613 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4662
2026-09-23T20:27:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1888 ms)
[----------] 18 tests from AuthTokenIssuingTest (34865 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:19 dist-test-slave-wmd0 krb5kdc[4760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:19 dist-test-slave-wmd0 krb5kdc[4760](info): set up 2 sockets
Sep 23 20:27:19 dist-test-slave-wmd0 krb5kdc[4760](info): commencing operation
krb5kdc: starting...
W20260923 20:27:20.403398 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:20 dist-test-slave-wmd0 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:27:20.658109 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:34075
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:34075,127.14.127.253:33625,127.14.127.252:40687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:20.989545 4773 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:27:20.990080 4773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:20.990195 4773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:20.990350 4773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:21.002444 4773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:21.002566 4773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:21.002693 4773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:21.002771 4773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:21.002867 4773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:21.018180 4773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:34075,127.14.127.253:33625,127.14.127.252:40687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:34075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:21.019706 4773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:21.021871 4773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:21.039211 4778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:21.039247 4781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:21.039306 4779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:21.040050 4773 server_base.cc:1034] running on GCE node
I20260923 20:27:21.040819 4773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:21.042177 4773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:21.043422 4773 hybrid_clock.cc:648] HybridClock initialized: now 1790195241043359 us; error 38 us; skew 500 ppm
I20260923 20:27:21.043756 4773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:21 dist-test-slave-wmd0 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195241, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:21.046793 4773 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:27:21.048208 4773 webserver.cc:460] Webserver started at http://127.14.127.254:35861/ using document root <none> and password file <none>
I20260923 20:27:21.048911 4773 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:21.049045 4773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:21.049357 4773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:21.052323 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "df8d8bc81def4f65b2f3ed187afa9c5a"
format_stamp: "Formatted at 2026-09-23 20:27:21 on dist-test-slave-wmd0"
server_key: "c090785ea7e0056975c54f9733d55ab2"
server_key_iv: "27e4d871db6532df44d8ea906eae2def"
server_key_version: "encryptionkey@0"
I20260923 20:27:21.053105 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "df8d8bc81def4f65b2f3ed187afa9c5a"
format_stamp: "Formatted at 2026-09-23 20:27:21 on dist-test-slave-wmd0"
server_key: "c090785ea7e0056975c54f9733d55ab2"
server_key_iv: "27e4d871db6532df44d8ea906eae2def"
server_key_version: "encryptionkey@0"
I20260923 20:27:21.058156 4773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:27:21.061544 4788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:21.062925 4773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:27:21.063107 4773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "df8d8bc81def4f65b2f3ed187afa9c5a"
format_stamp: "Formatted at 2026-09-23 20:27:21 on dist-test-slave-wmd0"
server_key: "c090785ea7e0056975c54f9733d55ab2"
server_key_iv: "27e4d871db6532df44d8ea906eae2def"
server_key_version: "encryptionkey@0"
I20260923 20:27:21.063338 4773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:21.082088 4773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:21.086073 4773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:21.086354 4773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:21.104808 4773 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:34075
I20260923 20:27:21.104830 4839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:34075 every 8 connection(s)
I20260923 20:27:21.106194 4773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:21.109822 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:21.112349 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4773
I20260923 20:27:21.112581 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:21.113332 14847 external_mini_cluster.cc:1428] Setting key eaba52748dca2f435fef65bd19ff7098
I20260923 20:27:21.119139 4840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:21 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195241, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.127.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.253@KRBTEST.COM" created.
I20260923 20:27:21.145243 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.122971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:34075 (local address 127.0.0.1:50942)
0923 20:27:21.123820 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.123872 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:21.126773 (+ 2901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:21.127637 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:21.127646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:21.128436 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:21.128966 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:21.128993 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.130495 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.130503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:21.132470 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:21.132488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.133390 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.133400 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:21.133648 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:21.134345 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:21.134380 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:21.137547 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:21.140842 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:21.140859 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:21.140873 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:21.141435 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:21.141927 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:21.141933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:21.141939 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:21.142171 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:21.143118 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:21.143124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:21.143360 (+ 236us) client_negotiation.cc:769] Sending connection context
0923 20:27:21.143577 (+ 217us) client_negotiation.cc:240] Negotiation successful
0923 20:27:21.144149 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":357,"threads_started":1}
I20260923 20:27:21.145282 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.124262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.14.127.254:34075)
0923 20:27:21.124954 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:21.124960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:21.125001 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:27:21.127155 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:21.127161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:21.127385 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:21.127537 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:21.129181 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.130335 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.132622 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.133145 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.133281 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:21.137838 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:21.137847 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:21.137859 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:21.137888 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:21.140697 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:21.141598 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:21.141607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:21.141612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:21.141812 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:21.142317 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:21.142322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:21.142327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:21.142980 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:21.143140 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:21.143726 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 20:27:21.144338 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":218,"server-negotiator.queue_time_us":478,"thread_start_us":348,"threads_started":1}
I20260923 20:27:21.148507 4840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:27:21.149679 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.149180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57022)
0923 20:27:21.149389 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.149526 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 20:27:21.150007 4791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
I20260923 20:27:21.151746 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.151276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57030)
0923 20:27:21.151450 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.151565 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:27:21.151980 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:33625: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab.
Entry for principal kudu/127.14.127.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab.
I20260923 20:27:21.173913 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.253:33625
--webserver_interface=127.14.127.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab
--principal=kudu/127.14.127.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:34075,127.14.127.253:33625,127.14.127.252:40687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:27:21.201364 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } attempt: 1
I20260923 20:27:21.202508 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.202055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57040)
0923 20:27:21.202254 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.202368 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:27:21.203508 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.203155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57054)
0923 20:27:21.203317 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.203383 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:27:21.203693 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:33625: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
I20260923 20:27:21.271981 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } attempt: 2
I20260923 20:27:21.273172 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.272673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57062)
0923 20:27:21.272929 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.273018 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:27:21.274211 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.273848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57072)
0923 20:27:21.274003 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.274088 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:27:21.274415 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:33625: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
I20260923 20:27:21.365698 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } attempt: 3
I20260923 20:27:21.366879 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.366392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57074)
0923 20:27:21.366602 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.366728 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:27:21.367904 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.367553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57084)
0923 20:27:21.367699 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.367771 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:27:21.368148 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:33625: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
W20260923 20:27:21.500427 4847 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:27:21.500804 4847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:21.500927 4847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:21.501087 4847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:21.508383 4847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:21.508477 4847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:21.508600 4847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:21.508692 4847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:21.508782 4847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:21.511408 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } attempt: 4
I20260923 20:27:21.512566 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.512019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57096)
0923 20:27:21.512333 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.512430 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260923 20:27:21.513585 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.513238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57098)
0923 20:27:21.513381 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.513469 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:27:21.513767 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:33625: Network error: Client connection negotiation failed: client connection to 127.14.127.253:33625: connect: Connection refused (error 111)
I20260923 20:27:21.519048 4847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:34075,127.14.127.253:33625,127.14.127.252:40687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab
--principal=kudu/127.14.127.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.253:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.127.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:21.520398 4847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:21.521729 4847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:21.533412 4852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:21.533381 4853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:21.533332 4855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:21.534281 4847 server_base.cc:1034] running on GCE node
I20260923 20:27:21.535120 4847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:21.536576 4847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:21.537819 4847 hybrid_clock.cc:648] HybridClock initialized: now 1790195241537762 us; error 35 us; skew 500 ppm
I20260923 20:27:21.538157 4847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:21 dist-test-slave-wmd0 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195241, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:21.541263 4847 init.cc:377] Logged in from keytab as kudu/127.14.127.253@KRBTEST.COM (short username kudu)
I20260923 20:27:21.542729 4847 webserver.cc:460] Webserver started at http://127.14.127.253:43163/ using document root <none> and password file <none>
I20260923 20:27:21.543421 4847 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:21.543556 4847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:21.543870 4847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:21.546958 4847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/data/instance:
uuid: "2377109e14824d968a6e65e25d2d8b80"
format_stamp: "Formatted at 2026-09-23 20:27:21 on dist-test-slave-wmd0"
server_key: "b041471c9ed8dfeb6384fe925b2dfe42"
server_key_iv: "91f7a94f75b7bd36578416cf489759e5"
server_key_version: "encryptionkey@0"
I20260923 20:27:21.547765 4847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/wal/instance:
uuid: "2377109e14824d968a6e65e25d2d8b80"
format_stamp: "Formatted at 2026-09-23 20:27:21 on dist-test-slave-wmd0"
server_key: "b041471c9ed8dfeb6384fe925b2dfe42"
server_key_iv: "91f7a94f75b7bd36578416cf489759e5"
server_key_version: "encryptionkey@0"
I20260923 20:27:21.552883 4847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:27:21.556448 4862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:21.557768 4847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:27:21.557956 4847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/wal
uuid: "2377109e14824d968a6e65e25d2d8b80"
format_stamp: "Formatted at 2026-09-23 20:27:21 on dist-test-slave-wmd0"
server_key: "b041471c9ed8dfeb6384fe925b2dfe42"
server_key_iv: "91f7a94f75b7bd36578416cf489759e5"
server_key_version: "encryptionkey@0"
I20260923 20:27:21.558182 4847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:21.583699 4847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:21.587620 4847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:21.587891 4847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:21.606303 4847 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.253:33625
I20260923 20:27:21.606323 4913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.253:33625 every 8 connection(s)
I20260923 20:27:21.607682 4847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
I20260923 20:27:21.611493 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:21.617662 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4847
I20260923 20:27:21.617872 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-1/wal/instance
I20260923 20:27:21.618294 14847 external_mini_cluster.cc:1428] Setting key 9a6b6d36b4f2f5c149aed4b87107d468
I20260923 20:27:21.617724 4914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:21 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195241, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:27:21.637672 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.620451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.14.127.254:34075)
0923 20:27:21.620633 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:21.620638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:21.621989 (+ 1351us) server_negotiation.cc:407] Connection header received
0923 20:27:21.623902 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:21.623908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:21.624004 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:21.624121 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:21.625560 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.626238 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.627915 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.628503 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.628616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:21.631992 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:21.631997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:21.632000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:21.632032 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:21.634762 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:21.635465 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:21.635471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:21.635476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:21.635598 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:21.635997 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:21.636002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:21.636006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:21.636287 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:21.636441 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:21.637126 (+ 685us) server_negotiation.cc:299] Negotiation successful
0923 20:27:21.637345 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:27:21.638751 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.621077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:34075 (local address 127.0.0.1:50948)
0923 20:27:21.621821 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.621858 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:21.623629 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:21.624352 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:21.624360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:21.624857 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:21.625394 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:21.625409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.626402 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.626406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:21.627727 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:21.627745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.628743 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.628749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:21.628979 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:21.629644 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:21.629666 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:21.631778 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:21.634900 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:21.634909 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:21.634924 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:21.635340 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:21.635708 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:21.635713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:21.635716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:21.635880 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:21.636416 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:21.636424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:21.636761 (+ 337us) client_negotiation.cc:769] Sending connection context
0923 20:27:21.637010 (+ 249us) client_negotiation.cc:240] Negotiation successful
0923 20:27:21.637502 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":319,"threads_started":1}
I20260923 20:27:21.640623 4914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.127.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.252@KRBTEST.COM" created.
Sep 23 20:27:21 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195241, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:27:21.654724 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.641442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57112)
0923 20:27:21.641644 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.641672 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:21.641797 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:21.642691 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:21.642694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:21.643019 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:21.643194 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:21.643202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.644603 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.644606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:21.645624 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:21.645636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.646415 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.646420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:21.646701 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:21.647274 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:21.647289 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:21.649197 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:21.652289 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:21.652296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:21.652301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:21.652642 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:21.652995 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:21.653000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:21.653004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:21.653153 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:21.653734 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:21.653739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:21.653882 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 20:27:21.654020 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 20:27:21.654307 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":41}
I20260923 20:27:21.654807 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.641446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.14.127.253:33625)
0923 20:27:21.642120 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:21.642129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:21.642163 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:27:21.642220 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:21.642228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:21.642488 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:21.642681 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:21.643374 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.644473 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.645753 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.646275 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.646408 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:21.649398 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:21.649408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:21.649420 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:21.649447 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:21.652155 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:21.652771 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:21.652785 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:21.652787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:21.652890 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:21.653252 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:21.653259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:21.653262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:21.653600 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:21.653810 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:21.654157 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 20:27:21.654411 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":459,"thread_start_us":303,"threads_started":1}
I20260923 20:27:21.657438 4914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:27:21.658686 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.658081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:52988)
0923 20:27:21.658316 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.658457 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 20:27:21.658946 4865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
I20260923 20:27:21.660605 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.660157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53000)
0923 20:27:21.660357 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.660457 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 20:27:21.660851 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab.
Entry for principal kudu/127.14.127.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab.
I20260923 20:27:21.676203 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.252:40687
--webserver_interface=127.14.127.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab
--principal=kudu/127.14.127.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:34075,127.14.127.253:33625,127.14.127.252:40687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:27:21.710204 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 1
I20260923 20:27:21.711500 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.710919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53016)
0923 20:27:21.711167 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.711319 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 20:27:21.712493 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.712132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53030)
0923 20:27:21.712280 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.712369 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:27:21.712707 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
I20260923 20:27:21.780932 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 2
I20260923 20:27:21.782227 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.781624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53034)
0923 20:27:21.781869 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.782001 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:27:21.783273 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.782892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53038)
0923 20:27:21.783046 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.783129 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:27:21.783496 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
I20260923 20:27:21.813084 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } attempt: 5
Sep 23 20:27:21 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195241, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:27:21.826344 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.814163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.14.127.253:33625)
0923 20:27:21.814385 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:21.814389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:21.814403 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:27:21.814469 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:21.814473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:21.814529 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:21.814663 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:21.815439 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.816121 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.817285 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.817733 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.817864 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:21.821232 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:21.821236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:21.821239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:21.821271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:21.823789 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:21.824565 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:21.824568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:21.824571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:21.824683 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:21.825077 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:21.825080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:21.825082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:21.825335 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:21.825442 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:21.825885 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 20:27:21.826123 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:27:21.826732 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.813877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57128)
0923 20:27:21.814074 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.814096 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:21.814237 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:21.814706 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:21.814713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:21.815108 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:21.815278 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:21.815287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.816259 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.816268 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:21.817130 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:21.817141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:21.817887 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:21.817893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:21.818084 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:21.818797 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:21.818815 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:21.821028 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:21.823944 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:21.823950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:21.823954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:21.824455 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:21.824795 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:21.824800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:21.824803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:21.824971 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:21.825456 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:21.825461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:21.825644 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 20:27:21.825806 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 20:27:21.826230 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:27:21.827906 4840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:27:21.828878 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.828508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53054)
0923 20:27:21.828684 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.828752 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 20:27:21.829730 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.829413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53070)
0923 20:27:21.829559 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.829641 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:27:21.829936 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
I20260923 20:27:21.874725 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 3
I20260923 20:27:21.875939 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.875406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53080)
0923 20:27:21.875632 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.875744 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:27:21.877024 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.876559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53090)
0923 20:27:21.876798 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.876890 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260923 20:27:21.877260 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
I20260923 20:27:21.881137 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 1
I20260923 20:27:21.882189 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.881686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53102)
0923 20:27:21.881889 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.881987 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 20:27:21.883332 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.882957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53112)
0923 20:27:21.883122 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.883206 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:27:21.883595 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
I20260923 20:27:21.951812 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 2
I20260923 20:27:21.953154 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.952676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53116)
0923 20:27:21.952908 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.953002 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:27:21.954164 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:21.953813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53126)
0923 20:27:21.953973 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:21.954049 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:27:21.954368 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
W20260923 20:27:22.009584 4921 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:27:22.009985 4921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:22.010124 4921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:22.010267 4921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:22.020474 4921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:22.020602 4921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260923 20:27:22.020479 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 4
W20260923 20:27:22.020697 4921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:22.020768 4921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:22.020859 4921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:22.021701 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.021163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53138)
0923 20:27:22.021404 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.021524 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:27:22.022694 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.022330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53146)
0923 20:27:22.022473 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.022551 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 20:27:22.022884 4914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
I20260923 20:27:22.032140 4921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:34075,127.14.127.253:33625,127.14.127.252:40687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab
--principal=kudu/127.14.127.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.252:40687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.127.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:22.033449 4921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:22.034883 4921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:22.046540 4927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:22.046572 4929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:22.046627 4926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:22.047364 4921 server_base.cc:1034] running on GCE node
I20260923 20:27:22.048159 4921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:22.049530 4921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:22.050722 4921 hybrid_clock.cc:648] HybridClock initialized: now 1790195242050655 us; error 57 us; skew 500 ppm
I20260923 20:27:22.051064 4921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195242, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:22.054296 4921 init.cc:377] Logged in from keytab as kudu/127.14.127.252@KRBTEST.COM (short username kudu)
I20260923 20:27:22.055868 4921 webserver.cc:460] Webserver started at http://127.14.127.252:38795/ using document root <none> and password file <none>
I20260923 20:27:22.056573 4921 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:22.056663 4921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:22.056916 4921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:22.059897 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/data/instance:
uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8"
format_stamp: "Formatted at 2026-09-23 20:27:22 on dist-test-slave-wmd0"
server_key: "11767fd5fe0f759a26735cf63258007c"
server_key_iv: "c4ae47f7510a9a9364f0578f4df678e3"
server_key_version: "encryptionkey@0"
I20260923 20:27:22.060551 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 3
I20260923 20:27:22.060798 4921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/wal/instance:
uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8"
format_stamp: "Formatted at 2026-09-23 20:27:22 on dist-test-slave-wmd0"
server_key: "11767fd5fe0f759a26735cf63258007c"
server_key_iv: "c4ae47f7510a9a9364f0578f4df678e3"
server_key_version: "encryptionkey@0"
I20260923 20:27:22.061654 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.061195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53158)
0923 20:27:22.061399 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.061491 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 20:27:22.062707 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.062299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53168)
0923 20:27:22.062478 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.062553 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 20:27:22.062932 4840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:40687: Network error: Client connection negotiation failed: client connection to 127.14.127.252:40687: connect: Connection refused (error 111)
I20260923 20:27:22.066345 4921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:27:22.069998 4936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:22.071584 4921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:27:22.071727 4921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/wal
uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8"
format_stamp: "Formatted at 2026-09-23 20:27:22 on dist-test-slave-wmd0"
server_key: "11767fd5fe0f759a26735cf63258007c"
server_key_iv: "c4ae47f7510a9a9364f0578f4df678e3"
server_key_version: "encryptionkey@0"
I20260923 20:27:22.071904 4921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:22.092592 4921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:22.096403 4921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:22.096613 4921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:22.118726 4921 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.252:40687
I20260923 20:27:22.118674 4987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.252:40687 every 8 connection(s)
I20260923 20:27:22.120067 4921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
I20260923 20:27:22.121573 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 4921
I20260923 20:27:22.121771 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194882707199-14847-0/minicluster-data/master-2/wal/instance
I20260923 20:27:22.122313 14847 external_mini_cluster.cc:1428] Setting key 3b5c55ffd4255fb00c5976dc18722a56
I20260923 20:27:22.124815 4988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:22.131804 4988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195242, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:27:22.144977 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.124317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.14.127.252:40687)
0923 20:27:22.125163 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.125174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.125215 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:27:22.126822 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.126857 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.127232 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.127651 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.129123 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.130447 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.131743 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.132430 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.132647 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.137100 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.137110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.137124 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.137166 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.140706 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.141588 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.141594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.141601 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.141772 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.142179 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.142183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.142185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.142722 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.143047 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.143487 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.143891 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":370,"threads_started":1}
I20260923 20:27:22.149942 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.134432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.14.127.254:34075)
0923 20:27:22.134716 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.134723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.135389 (+ 666us) server_negotiation.cc:407] Connection header received
0923 20:27:22.135579 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.135585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.135665 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.135783 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.136902 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.137960 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.139611 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.140277 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.140412 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.143929 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.143934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.143937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.143972 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.146972 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.147829 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.147834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.147837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.148003 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.148450 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.148454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.148456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.148877 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.149038 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.149538 (+ 500us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.149697 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:27:22.150213 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.134363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:34075 (local address 127.0.0.1:50962)
0923 20:27:22.135232 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.135275 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:22.135435 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:22.135839 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:22.135843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:22.136502 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:22.136746 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.136755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.138132 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.138136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.139452 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.139464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.140463 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.140468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.140593 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.141172 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:22.141195 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:22.143594 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:22.147112 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.147121 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.147125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.147680 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.148132 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.148136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.148139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.148299 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.149009 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:22.149014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:22.149212 (+ 198us) client_negotiation.cc:769] Sending connection context
0923 20:27:22.149424 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 20:27:22.149763 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":460,"threads_started":1}
I20260923 20:27:22.151641 4988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195242, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:27:22.165632 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.148335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.14.127.254:34075)
0923 20:27:22.148966 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.148972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.148995 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:27:22.149038 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.149043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.149115 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.149190 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.150717 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.151674 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.153198 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.153600 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.153740 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.158685 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.158697 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.158701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.158733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.162935 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.163548 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.163554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.163557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.163700 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.164077 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.164080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.164082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.164415 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.164672 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.165013 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.165235 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":267,"threads_started":1}
I20260923 20:27:22.166324 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.152501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:33625 (local address 127.0.0.1:57130)
0923 20:27:22.152681 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.152699 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:22.152792 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:22.153101 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:22.153108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:22.153447 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:22.153586 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.153594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.154600 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.154629 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.156124 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.156144 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.156872 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.156876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.157014 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.157611 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:22.157634 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:22.160047 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:22.163696 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.163699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.163702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.164203 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.164588 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.164593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.164596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.164757 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.165432 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:22.165438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:22.165604 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 20:27:22.165762 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 20:27:22.166006 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:27:22.166522 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.152550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.14.127.253:33625)
0923 20:27:22.152728 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.152734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.152761 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:27:22.152905 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.152908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.152966 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.153045 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.153734 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.154449 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.156293 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.156702 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.156844 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.160238 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.160246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.160252 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.160291 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.163577 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.164313 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.164318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.164321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.164479 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.164884 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.164889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.164894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.165220 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.165359 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.165907 (+ 548us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.166147 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:27:22.167764 4988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195242, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:27:22.181263 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.168400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53186)
0923 20:27:22.168537 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.168559 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:22.168655 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:22.168951 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:22.168956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:22.169427 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:22.169593 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.169603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.170470 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.170474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.171460 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.171477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.172146 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.172154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.172347 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.172943 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:22.172959 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:22.175117 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:22.178583 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.178590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.178595 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.179094 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.179515 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.179520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.179526 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.179695 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.180356 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:22.180360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:22.180484 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 20:27:22.180620 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 20:27:22.180928 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 20:27:22.181597 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.168541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.14.127.252:40687)
0923 20:27:22.168694 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.168698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.168714 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:27:22.168748 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.168751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.168797 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.168866 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.169724 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.170367 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.171605 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.171992 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.172082 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.175275 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.175280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.175285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.175319 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.178421 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.179216 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.179221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.179225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.179369 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.179828 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.179834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.179840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.180204 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.180360 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.180768 (+ 408us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.181139 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":54}
I20260923 20:27:22.184042 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.167712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.14.127.253:33625)
0923 20:27:22.167863 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.167869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.167888 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:27:22.167937 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.167941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.168008 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.168120 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.170498 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.171246 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.172766 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.173235 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.173325 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.177852 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.177859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.177863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.177904 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.181227 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.182166 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.182170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.182172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.182302 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.182792 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.182797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.182799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.183096 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.183305 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.183667 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.183827 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 20:27:22.187309 4988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: Bootstrap starting.
I20260923 20:27:22.188995 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:22.190732 4988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:22.191943 4988 log.cc:824] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:22.195091 4988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: No bootstrap required, opened a new log
I20260923 20:27:22.191644 4815 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50966
I20260923 20:27:22.203207 4988 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } }
I20260923 20:27:22.203506 4988 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8929180ad114ab38ee4ddd7f8d6f7b8, State: Initialized, Role: FOLLOWER
I20260923 20:27:22.203881 4988 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } }
I20260923 20:27:22.204743 4995 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.205122 4995 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:22.205651 4988 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:27:22.215058 5006 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:27:22.215179 5006 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:27:22.212940 4921 master_runner.cc:186] Error getting master registration for 127.14.127.254:34075: OK, instance_id { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" instance_seqno: 1790195241099567 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:27:22.220768 4921 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:27:22.240139 4840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 4
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195241, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
I20260923 20:27:22.255126 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.240898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53202)
0923 20:27:22.241173 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.241193 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:22.241330 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:22.241694 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:22.241698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:22.242113 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:22.242268 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.242276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.243359 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.243364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.244295 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.244306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.245042 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.245047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.245258 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.245878 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:22.245896 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:22.248200 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:22.252357 (+ 4157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.252363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.252367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.252862 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.253266 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.253271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.253274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.253446 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.254090 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:22.254094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:22.254282 (+ 188us) client_negotiation.cc:769] Sending connection context
0923 20:27:22.254424 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 20:27:22.254720 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260923 20:27:22.255208 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.240946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.14.127.252:40687)
0923 20:27:22.241149 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.241156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.241289 (+ 133us) server_negotiation.cc:407] Connection header received
0923 20:27:22.241465 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.241473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.241539 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.241655 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.242420 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.243224 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.244438 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.244893 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.245036 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.248409 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.248416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.248422 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.248460 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.252212 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.252995 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.253001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.253005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.253147 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.253570 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.253576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.253579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.253949 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.254133 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.254561 (+ 428us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.254821 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:27:22.259676 4840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a: Bootstrap starting.
I20260923 20:27:22.263183 4840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:22.264219 4840 log.cc:824] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:22.267230 4840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a: No bootstrap required, opened a new log
I20260923 20:27:22.272384 4840 raft_consensus.cc:348] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } }
I20260923 20:27:22.272750 4840 raft_consensus.cc:729] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8d8bc81def4f65b2f3ed187afa9c5a, State: Initialized, Role: FOLLOWER
I20260923 20:27:22.273190 4840 consensus_queue.cc:260] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } }
I20260923 20:27:22.274137 5009 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.274593 5009 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:22.275410 4840 sys_catalog.cc:564] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:22.275581 4815 raft_consensus.cc:482] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 20:27:22.275791 4815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:22.280149 4815 raft_consensus.cc:504] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } }
I20260923 20:27:22.282886 4815 leader_election.cc:288] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [CANDIDATE]: Term 1 election: Requested vote from peers 2377109e14824d968a6e65e25d2d8b80 (127.14.127.253:33625), f8929180ad114ab38ee4ddd7f8d6f7b8 (127.14.127.252:40687)
I20260923 20:27:22.284031 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8"
I20260923 20:27:22.284499 4963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:22.288105 4963 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8d8bc81def4f65b2f3ed187afa9c5a in term 1.
I20260923 20:27:22.289199 4791 leader_election.cc:302] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df8d8bc81def4f65b2f3ed187afa9c5a, f8929180ad114ab38ee4ddd7f8d6f7b8; no voters:
I20260923 20:27:22.289939 5009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:22.293293 5009 raft_consensus.cc:686] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [term 1 LEADER]: Becoming Leader. State: Replica: df8d8bc81def4f65b2f3ed187afa9c5a, State: Running, Role: LEADER
I20260923 20:27:22.293911 5009 consensus_queue.cc:237] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } }
I20260923 20:27:22.300825 5018 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8d8bc81def4f65b2f3ed187afa9c5a. Latest consensus state: current_term: 1 leader_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.301283 5018 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:22.308605 5022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:22.322099 4914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } attempt: 5
I20260923 20:27:22.322474 5022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 20:27:22 dist-test-slave-wmd0 krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195241, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
W20260923 20:27:22.339318 4794 tablet.cc:2369] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:27:22.343323 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.323478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:40687 (local address 127.0.0.1:53218)
0923 20:27:22.323791 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:22.323814 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:22.323970 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:22.324614 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:22.324621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:22.325165 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:22.325383 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.325396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.326895 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.326901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.328350 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:22.328366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.329875 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.329881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:22.330126 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.330852 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:22.330882 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:22.334180 (+ 3298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:22.338714 (+ 4534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.338719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.338723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.340310 (+ 1587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.340998 (+ 688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:22.341002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:22.341006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:22.341222 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:22.342264 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:22.342269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:22.342434 (+ 165us) client_negotiation.cc:769] Sending connection context
0923 20:27:22.342598 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 20:27:22.342917 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260923 20:27:22.344532 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.323068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.14.127.252:40687)
0923 20:27:22.323292 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.323300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.324116 (+ 816us) server_negotiation.cc:407] Connection header received
0923 20:27:22.324202 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.324210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.324307 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.324391 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.325570 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.326701 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.328835 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.329538 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.329655 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.334377 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.334386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.334395 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.334447 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.338520 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.340467 (+ 1947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.340473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.340477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.340638 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.341420 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.341425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.341428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.341763 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.341909 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.343957 (+ 2048us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.344200 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 20:27:22.348523 4942 tablet.cc:2369] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:27:22.350896 4914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80: Bootstrap starting.
I20260923 20:27:22.351823 5030 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:27:22.354948 4914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:22.356281 4914 log.cc:824] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:22.362556 4773 master_runner.cc:186] Error getting master registration for 127.14.127.253:33625: OK, instance_id { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" instance_seqno: 1790195241601043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:27:22.377367 4963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [term 1 FOLLOWER]: Refusing update from remote peer df8d8bc81def4f65b2f3ed187afa9c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:27:22.378474 5018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:27:22.388974 4773 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:27:22.380875 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.321081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.14.127.254:34075)
0923 20:27:22.321298 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.321307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.327561 (+ 6254us) server_negotiation.cc:407] Connection header received
0923 20:27:22.327873 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.327881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.327972 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.328049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.334765 (+ 6716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.335981 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.340441 (+ 4460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.341319 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.341436 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.350640 (+ 9204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.350649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.350655 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.350695 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.356128 (+ 5433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.358625 (+ 2497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.358632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.358636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.358789 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.377810 (+ 19021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.377819 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.377822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.378294 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.378573 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.379758 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.380258 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 20:27:22.419891 4995 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8d8bc81def4f65b2f3ed187afa9c5a. Latest consensus state: current_term: 1 leader_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.420259 4995 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:22.423669 4995 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.424002 4995 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:22.425743 5018 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.426040 5018 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:22.428742 5022 catalog_manager.cc:1347] Generated new cluster ID: 535d7341298547f2912de5611d3aed3b
I20260923 20:27:22.428856 5022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:22.432721 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.320696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.14.127.253:33625)
0923 20:27:22.320906 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.320914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.351554 (+ 30640us) server_negotiation.cc:407] Connection header received
0923 20:27:22.351622 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.351629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.351709 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.351807 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.385543 (+ 33736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.386703 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.397197 (+ 10494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.397896 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.398001 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.413939 (+ 15938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.413947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.413952 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.414009 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.417845 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.423555 (+ 5710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.423562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.423565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.423743 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.427945 (+ 4202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.427954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.427960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.428433 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.431083 (+ 2650us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.432059 (+ 976us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.432323 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:27:22.433626 4914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80: No bootstrap required, opened a new log
I20260923 20:27:22.435797 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.331226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.14.127.252:40687)
0923 20:27:22.347291 (+ 16065us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.347298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.355022 (+ 7724us) server_negotiation.cc:407] Connection header received
0923 20:27:22.355809 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.355816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.355924 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.356000 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.381357 (+ 25357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.382465 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.391639 (+ 9174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.392423 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.392542 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.407212 (+ 14670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:22.407220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:22.407226 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:22.407273 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:22.411600 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.421070 (+ 9470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.421076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.421080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.421245 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:22.424380 (+ 3135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:22.424386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:22.424392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:22.424787 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:22.425077 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.433065 (+ 7988us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.435280 (+ 2215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15937,"thread_start_us":23358,"threads_started":1}
I20260923 20:27:22.445922 4914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } }
I20260923 20:27:22.446498 4914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2377109e14824d968a6e65e25d2d8b80, State: Initialized, Role: FOLLOWER
I20260923 20:27:22.447191 4914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } }
I20260923 20:27:22.448637 5035 mvcc.cc:205] Tried to move back new op lower bound from 7332639712662552576 to 7332639712444100608. Current Snapshot: MvccSnapshot[applied={T|T < 7332639712662552576}]
I20260923 20:27:22.453028 5022 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:22.460487 5022 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:22.465320 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.465942 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:22.466782 4914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:22.468891 4888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:22.472668 4888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [term 1 FOLLOWER]: Refusing update from remote peer df8d8bc81def4f65b2f3ed187afa9c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:27:22.473388 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2377109e14824d968a6e65e25d2d8b80"
I20260923 20:27:22.473495 5038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:27:22.477254 4889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8d8bc81def4f65b2f3ed187afa9c5a in term 1.
I20260923 20:27:22.497071 5022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a: Generated new TSK 0
I20260923 20:27:22.498422 5022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:22.503540 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df8d8bc81def4f65b2f3ed187afa9c5a. Latest consensus state: current_term: 1 leader_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.504027 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:22.504642 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.505169 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:22.506417 5009 sys_catalog.cc:455] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8d8bc81def4f65b2f3ed187afa9c5a" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 34075 } } peers { permanent_uuid: "2377109e14824d968a6e65e25d2d8b80" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 33625 } } peers { permanent_uuid: "f8929180ad114ab38ee4ddd7f8d6f7b8" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 40687 } } }
I20260923 20:27:22.506755 5009 sys_catalog.cc:458] T 00000000000000000000000000000000 P df8d8bc81def4f65b2f3ed187afa9c5a [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:22.509162 5044 mvcc.cc:205] Tried to move back new op lower bound from 7332639713063440384 to 7332639712444100608. Current Snapshot: MvccSnapshot[applied={T|T < 7332639712662552576}]
I20260923 20:27:22.606212 5060 catalog_manager.cc:1259] Loaded cluster ID: 535d7341298547f2912de5611d3aed3b
I20260923 20:27:22.606463 5060 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80: loading cluster ID for follower catalog manager: success
I20260923 20:27:22.610553 5060 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80: acquiring CA information for follower catalog manager: success
I20260923 20:27:22.613116 5060 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2377109e14824d968a6e65e25d2d8b80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 20:27:22.625447 4993 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.607407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.14.127.254:34075)
0923 20:27:22.607627 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.607633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.623208 (+ 15575us) server_negotiation.cc:407] Connection header received
0923 20:27:22.623516 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.623523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.623613 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.623696 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.625121 (+ 1425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50984: BlockingRecv error: recv got EOF from 127.0.0.1:50984 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 20:27:22.628463 4918 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.613622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.14.127.253:33625)
0923 20:27:22.613812 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.613819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.626337 (+ 12518us) server_negotiation.cc:407] Connection header received
0923 20:27:22.626743 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.626752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.626898 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.626991 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.628136 (+ 1145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57164: BlockingRecv error: recv got EOF from 127.0.0.1:57164 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 20:27:22.631686 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.623368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.14.127.252:40687)
0923 20:27:22.623603 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.623609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.629356 (+ 5747us) server_negotiation.cc:407] Connection header received
0923 20:27:22.629643 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.629652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.629747 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.629826 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.631365 (+ 1539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53234: BlockingRecv error: recv got EOF from 127.0.0.1:53234 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:27:22.679242 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.660140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.14.127.253:33625)
0923 20:27:22.660328 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.660335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.671191 (+ 10856us) server_negotiation.cc:407] Connection header received
0923 20:27:22.671817 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.671824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.672051 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.672188 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:22.672871 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.674044 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.676133 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.676811 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.676906 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.677698 (+ 792us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:22.678385 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:22.678506 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.678783 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.678881 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 20:27:22.725181 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.715381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.14.127.252:40687)
0923 20:27:22.715658 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.715664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.723060 (+ 7396us) server_negotiation.cc:407] Connection header received
0923 20:27:22.723572 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.723578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.723711 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.723787 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.724903 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53248: BlockingRecv error: recv got EOF from 127.0.0.1:53248 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:27:22.779038 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.763766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.14.127.253:33625)
0923 20:27:22.763974 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.763981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.771762 (+ 7781us) server_negotiation.cc:407] Connection header received
0923 20:27:22.772270 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.772276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.772394 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.772462 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:22.773170 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.774373 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.776649 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.777387 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.777490 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.778149 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:22.778308 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:22.778373 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.778627 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.778714 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 20:27:22.822150 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.812580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.14.127.252:40687)
0923 20:27:22.812770 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.812777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.820148 (+ 7371us) server_negotiation.cc:407] Connection header received
0923 20:27:22.820659 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.820665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.820769 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.820844 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.821881 (+ 1037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53250: BlockingRecv error: recv got EOF from 127.0.0.1:53250 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:27:22.874332 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.859382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.14.127.253:33625)
0923 20:27:22.859565 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.859573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.867333 (+ 7760us) server_negotiation.cc:407] Connection header received
0923 20:27:22.867920 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.867926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.868024 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.868087 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:22.868741 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.869897 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.872008 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.872696 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.872813 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.873455 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:22.873615 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:22.873678 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.873898 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.873995 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 20:27:22.910739 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.903206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.14.127.252:40687)
0923 20:27:22.903386 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.903393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.908989 (+ 5596us) server_negotiation.cc:407] Connection header received
0923 20:27:22.909205 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.909211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.909322 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.909462 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.910452 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53264: BlockingRecv error: recv got EOF from 127.0.0.1:53264 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:27:22.929876 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.924228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.14.127.253:33625)
0923 20:27:22.924443 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.924450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.925034 (+ 584us) server_negotiation.cc:407] Connection header received
0923 20:27:22.925323 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.925331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.925410 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.925510 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:22.925930 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.926773 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.928107 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.928549 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.928687 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.929072 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:22.929168 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:22.929262 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.929434 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.929514 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260923 20:27:22.938448 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.936852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.14.127.252:40687)
0923 20:27:22.937124 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.937130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.937150 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:27:22.937361 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.937366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.937454 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.937563 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.938269 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53278: BlockingRecv error: recv got EOF from 127.0.0.1:53278 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
I20260923 20:27:22.955829 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.950341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.14.127.253:33625)
0923 20:27:22.950603 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.950638 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.951158 (+ 520us) server_negotiation.cc:407] Connection header received
0923 20:27:22.951381 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.951387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.951473 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.951579 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:22.951970 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.952688 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.953988 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.954429 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.954566 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.955036 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:22.955150 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:22.955251 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.955406 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.955486 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 20:27:22.964279 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.962005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.14.127.252:40687)
0923 20:27:22.962241 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.962250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.962831 (+ 581us) server_negotiation.cc:407] Connection header received
0923 20:27:22.963047 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.963052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.963137 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.963246 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.964024 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53280: BlockingRecv error: recv got EOF from 127.0.0.1:53280 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:27:22.989907 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:22.983779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.14.127.253:33625)
0923 20:27:22.984026 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.984033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.984582 (+ 549us) server_negotiation.cc:407] Connection header received
0923 20:27:22.984813 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.984817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.984887 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.984992 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:22.985403 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.986196 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.987635 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:22.988159 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:22.988286 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:22.989053 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:22.989230 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:22.989285 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:22.989481 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 20:27:22.989583 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260923 20:27:22.998391 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:22.996093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.14.127.252:40687)
0923 20:27:22.996326 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:22.996335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:22.996881 (+ 546us) server_negotiation.cc:407] Connection header received
0923 20:27:22.997207 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:22.997213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:22.997312 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:22.997425 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:22.998201 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53282: BlockingRecv error: recv got EOF from 127.0.0.1:53282 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:27:23.040299 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:23.034437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.14.127.253:33625)
0923 20:27:23.034682 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.034688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.034995 (+ 307us) server_negotiation.cc:407] Connection header received
0923 20:27:23.035174 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.035180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.035279 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.035401 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:23.035815 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.037000 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.038257 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.039013 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.039147 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:23.039505 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:23.039643 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:23.039733 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:23.039870 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 20:27:23.039953 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 20:27:23.048206 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:23.046207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.14.127.252:40687)
0923 20:27:23.046397 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.046403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.046826 (+ 423us) server_negotiation.cc:407] Connection header received
0923 20:27:23.047020 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.047026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.047121 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.047224 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:23.047970 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53286: BlockingRecv error: recv got EOF from 127.0.0.1:53286 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:27:23.124148 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:23.116178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.14.127.253:33625)
0923 20:27:23.116367 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.116371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.116510 (+ 139us) server_negotiation.cc:407] Connection header received
0923 20:27:23.116746 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.116755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.116840 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.116952 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:23.117472 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.119723 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.121753 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.122473 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.122585 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:23.123247 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:23.123396 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:23.123463 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:23.123746 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 20:27:23.123842 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 20:27:23.143105 4991 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:23.140856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.14.127.252:40687)
0923 20:27:23.141052 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.141060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.141596 (+ 536us) server_negotiation.cc:407] Connection header received
0923 20:27:23.141792 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.141798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.141889 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.141993 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:23.142876 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53298: BlockingRecv error: recv got EOF from 127.0.0.1:53298 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:27:23.217248 5006 catalog_manager.cc:1259] Loaded cluster ID: 535d7341298547f2912de5611d3aed3b
I20260923 20:27:23.217396 5006 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: loading cluster ID for follower catalog manager: success
I20260923 20:27:23.219913 5006 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: acquiring CA information for follower catalog manager: success
I20260923 20:27:23.221681 5006 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8929180ad114ab38ee4ddd7f8d6f7b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:27:23.279933 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:23.275005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.14.127.253:33625)
0923 20:27:23.275269 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.275277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.275444 (+ 167us) server_negotiation.cc:407] Connection header received
0923 20:27:23.275654 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.275660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.275740 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.275826 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:23.276254 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.277076 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.278317 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.278769 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.278859 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:23.279243 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:23.279372 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:23.279454 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:23.279631 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 20:27:23.279698 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:27:23.291263 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:23.285853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.14.127.252:40687)
0923 20:27:23.286081 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.286087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.286287 (+ 200us) server_negotiation.cc:407] Connection header received
0923 20:27:23.286423 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.286426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.286575 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.286727 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:23.287097 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.287896 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.289132 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.289534 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.289630 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:23.290079 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:23.290670 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:23.290800 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:23.290865 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 20:27:23.290928 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 20:27:23.305846 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:23.297434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.14.127.254:34075)
0923 20:27:23.298133 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.298139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.298157 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:27:23.298480 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.298486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.298739 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.298928 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:23.299912 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.301033 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.303549 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.304089 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.304234 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:23.304719 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:23.305168 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:23.305284 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:23.305420 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 20:27:23.305506 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":697,"threads_started":1}
I20260923 20:27:23.305874 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:23.297956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.14.127.253:33625)
0923 20:27:23.298130 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.298136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.298746 (+ 610us) server_negotiation.cc:407] Connection header received
0923 20:27:23.299042 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.299048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.299151 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.299238 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:23.299975 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.301037 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.303584 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.304191 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.304308 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:23.304892 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:23.305032 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:23.305114 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:23.305554 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:27:23.305632 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:27:23.305846 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:23.298849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.14.127.252:40687)
0923 20:27:23.299065 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:23.299073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:23.300472 (+ 1399us) server_negotiation.cc:407] Connection header received
0923 20:27:23.300621 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:23.300626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:23.300699 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:23.300787 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:23.301477 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.302154 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.303748 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:23.304461 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:23.304614 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:23.305098 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:23.305215 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:23.305291 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:23.305459 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 20:27:23.305537 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:27:23.311406 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4773
I20260923 20:27:23.312389 4835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:23.422955 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4773
I20260923 20:27:23.444499 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4847
I20260923 20:27:23.445585 4909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:23.552656 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4847
I20260923 20:27:23.573088 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 4921
I20260923 20:27:23.574229 4983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:23.680539 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 4921
2026-09-23T20:27:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4388 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:27:23 dist-test-slave-wmd0 krb5kdc[5172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:27:23 dist-test-slave-wmd0 krb5kdc[5172](info): set up 2 sockets
Sep 23 20:27:23 dist-test-slave-wmd0 krb5kdc[5172](info): commencing operation
krb5kdc: starting...
W20260923 20:27:24.783268 14847 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/test-user.keytab.
Sep 23 20:27:24 dist-test-slave-wmd0 krb5kdc[5172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.127.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
Entry for principal kudu/127.14.127.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab.
I20260923 20:27:25.041610 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:40631
--webserver_interface=127.14.127.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:40631,127.14.127.253:45227,127.14.127.252:39459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:27:25.368021 5185 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:27:25.368420 5185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:25.368549 5185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:25.368685 5185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:25.376533 5185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:25.376633 5185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:25.376739 5185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:25.376832 5185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:25.376961 5185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:25.387086 5185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:40631,127.14.127.253:45227,127.14.127.252:39459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.254.keytab
--principal=kudu/127.14.127.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.254:40631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.127.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:25.388486 5185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:25.389804 5185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:25.402992 5193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:25.403137 5191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:25.403421 5190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:25.403826 5185 server_base.cc:1034] running on GCE node
I20260923 20:27:25.404609 5185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:25.406047 5185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:25.407265 5185 hybrid_clock.cc:648] HybridClock initialized: now 1790195245407206 us; error 31 us; skew 500 ppm
I20260923 20:27:25.407680 5185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:25 dist-test-slave-wmd0 krb5kdc[5172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195245, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:25.411232 5185 init.cc:377] Logged in from keytab as kudu/127.14.127.254@KRBTEST.COM (short username kudu)
I20260923 20:27:25.412748 5185 webserver.cc:460] Webserver started at http://127.14.127.254:33239/ using document root <none> and password file <none>
I20260923 20:27:25.413477 5185 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:25.413615 5185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:25.413944 5185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:25.417214 5185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/data/instance:
uuid: "9e9b743cc98a423788792c2ef7fd453b"
format_stamp: "Formatted at 2026-09-23 20:27:25 on dist-test-slave-wmd0"
server_key: "92e42e9d2932a89a2e1a1054b6ca3153"
server_key_iv: "7ce12457cce1fc5260cccd268cebeca8"
server_key_version: "encryptionkey@0"
I20260923 20:27:25.418047 5185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/wal/instance:
uuid: "9e9b743cc98a423788792c2ef7fd453b"
format_stamp: "Formatted at 2026-09-23 20:27:25 on dist-test-slave-wmd0"
server_key: "92e42e9d2932a89a2e1a1054b6ca3153"
server_key_iv: "7ce12457cce1fc5260cccd268cebeca8"
server_key_version: "encryptionkey@0"
I20260923 20:27:25.423914 5185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:27:25.427664 5200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:25.429059 5185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:27:25.429277 5185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/wal
uuid: "9e9b743cc98a423788792c2ef7fd453b"
format_stamp: "Formatted at 2026-09-23 20:27:25 on dist-test-slave-wmd0"
server_key: "92e42e9d2932a89a2e1a1054b6ca3153"
server_key_iv: "7ce12457cce1fc5260cccd268cebeca8"
server_key_version: "encryptionkey@0"
I20260923 20:27:25.429517 5185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:25.442313 5185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:25.446740 5185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:25.447031 5185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:25.465612 5185 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.254:40631
I20260923 20:27:25.465631 5251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.254:40631 every 8 connection(s)
I20260923 20:27:25.467121 5185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/data/info.pb
I20260923 20:27:25.470919 5252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:25.476383 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 5185
I20260923 20:27:25.476579 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-0/wal/instance
I20260923 20:27:25.477025 14847 external_mini_cluster.cc:1428] Setting key b8ce04b7031882b004303a7e9ce01b79
I20260923 20:27:25.477581 5252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:25 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195245, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:27:25.500991 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.482221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.14.127.254:40631)
0923 20:27:25.482968 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:25.482977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:25.483031 (+ 54us) server_negotiation.cc:407] Connection header received
0923 20:27:25.484510 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:25.484516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:25.484758 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:25.484965 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:25.486263 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:25.487535 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:25.489006 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:25.489566 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:25.489768 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:25.493533 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:25.493541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:25.493553 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:25.493587 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:25.496379 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:25.497368 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:25.497378 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:25.497381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:25.497562 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:25.498084 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:25.498088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:25.498090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:25.498595 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:25.498841 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:25.499562 (+ 721us) server_negotiation.cc:299] Negotiation successful
0923 20:27:25.499904 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":532,"thread_start_us":378,"threads_started":1}
I20260923 20:27:25.501032 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.480928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:40631 (local address 127.0.0.1:56270)
0923 20:27:25.481708 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.481754 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:25.484106 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:25.484992 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:25.484999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:25.485704 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:25.486045 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:25.486062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:25.487697 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:25.487705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:25.488850 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:25.488863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:25.489804 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:25.489811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:25.490027 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:25.490757 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:25.490779 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:25.493294 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:25.496533 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:25.496549 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:25.496564 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:25.497206 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:25.497689 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:25.497695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:25.497699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:25.497940 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:25.498819 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:25.498829 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:25.499130 (+ 301us) client_negotiation.cc:769] Sending connection context
0923 20:27:25.499446 (+ 316us) client_negotiation.cc:240] Negotiation successful
0923 20:27:25.500115 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":122,"thread_start_us":323,"threads_started":1}
WARNING: no policy specified for kudu/127.14.127.253@KRBTEST.COM; defaulting to no policy
I20260923 20:27:25.504374 5252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.253@KRBTEST.COM" created.
I20260923 20:27:25.505595 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.505215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45666)
0923 20:27:25.505384 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.505488 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 20:27:25.505906 5201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
I20260923 20:27:25.507493 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.507157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45668)
0923 20:27:25.507303 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.507373 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:27:25.507730 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45227: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab.
Entry for principal kudu/127.14.127.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab.
I20260923 20:27:25.534085 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.253:45227
--webserver_interface=127.14.127.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab
--principal=kudu/127.14.127.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:40631,127.14.127.253:45227,127.14.127.252:39459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:27:25.557118 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } attempt: 1
I20260923 20:27:25.558471 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.557824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45684)
0923 20:27:25.558091 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.558260 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 20:27:25.559569 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.559157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45694)
0923 20:27:25.559307 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.559403 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:27:25.559774 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45227: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
I20260923 20:27:25.628062 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } attempt: 2
I20260923 20:27:25.629410 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.628811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45708)
0923 20:27:25.629103 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.629215 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 20:27:25.630695 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.630187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45710)
0923 20:27:25.630351 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.630476 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 20:27:25.630915 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45227: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
I20260923 20:27:25.722205 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } attempt: 3
I20260923 20:27:25.723659 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.723000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45726)
0923 20:27:25.723307 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.723459 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260923 20:27:25.724864 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.724388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45732)
0923 20:27:25.724566 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.724667 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:27:25.725090 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45227: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
W20260923 20:27:25.864759 5259 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:27:25.865075 5259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:25.865209 5259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:25.865365 5259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:27:25.868353 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } attempt: 4
I20260923 20:27:25.869606 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.868987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45746)
0923 20:27:25.869320 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.869425 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260923 20:27:25.870658 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.870227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45748)
0923 20:27:25.870378 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.870479 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 20:27:25.870843 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.253:45227: Network error: Client connection negotiation failed: client connection to 127.14.127.253:45227: connect: Connection refused (error 111)
W20260923 20:27:25.873445 5259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:25.873522 5259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:25.873610 5259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:25.873689 5259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:25.873787 5259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:25.883852 5259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:40631,127.14.127.253:45227,127.14.127.252:39459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.253.keytab
--principal=kudu/127.14.127.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.253:45227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.127.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:25.885162 5259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:25.886461 5259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:25.898509 5267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:25.898594 5264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:25.898571 5265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:25.900116 5259 server_base.cc:1034] running on GCE node
I20260923 20:27:25.900831 5259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:25.902204 5259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:25.903473 5259 hybrid_clock.cc:648] HybridClock initialized: now 1790195245903402 us; error 37 us; skew 500 ppm
I20260923 20:27:25.903827 5259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:27:25 dist-test-slave-wmd0 krb5kdc[5172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195245, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:25.906922 5259 init.cc:377] Logged in from keytab as kudu/127.14.127.253@KRBTEST.COM (short username kudu)
I20260923 20:27:25.908355 5259 webserver.cc:460] Webserver started at http://127.14.127.253:40617/ using document root <none> and password file <none>
I20260923 20:27:25.909088 5259 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:25.909219 5259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:25.909549 5259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:25.912621 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/data/instance:
uuid: "dfee02eb173141f3907249cf9f13d36d"
format_stamp: "Formatted at 2026-09-23 20:27:25 on dist-test-slave-wmd0"
server_key: "04d8d1fd360880cbe00a05a64cc421e6"
server_key_iv: "eb17955d4067e4573d55b6a62444a864"
server_key_version: "encryptionkey@0"
I20260923 20:27:25.913414 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/wal/instance:
uuid: "dfee02eb173141f3907249cf9f13d36d"
format_stamp: "Formatted at 2026-09-23 20:27:25 on dist-test-slave-wmd0"
server_key: "04d8d1fd360880cbe00a05a64cc421e6"
server_key_iv: "eb17955d4067e4573d55b6a62444a864"
server_key_version: "encryptionkey@0"
I20260923 20:27:25.918785 5259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 20:27:25.922361 5274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:25.923815 5259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:27:25.924005 5259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/wal
uuid: "dfee02eb173141f3907249cf9f13d36d"
format_stamp: "Formatted at 2026-09-23 20:27:25 on dist-test-slave-wmd0"
server_key: "04d8d1fd360880cbe00a05a64cc421e6"
server_key_iv: "eb17955d4067e4573d55b6a62444a864"
server_key_version: "encryptionkey@0"
I20260923 20:27:25.924238 5259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:25.953599 5259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:25.957684 5259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:25.957983 5259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:25.976598 5259 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.253:45227
I20260923 20:27:25.976613 5325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.253:45227 every 8 connection(s)
I20260923 20:27:25.977993 5259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/data/info.pb
I20260923 20:27:25.979570 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 5259
I20260923 20:27:25.979763 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-1/wal/instance
I20260923 20:27:25.980333 14847 external_mini_cluster.cc:1428] Setting key 2ef2fbd71c22aae1ca202f8c66ee0bcc
I20260923 20:27:25.982717 5326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:25.992683 5326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.127.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.127.252@KRBTEST.COM" created.
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195245, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:27:26.031818 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.995503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.14.127.254:40631)
0923 20:27:25.995730 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:25.995737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:25.997017 (+ 1280us) server_negotiation.cc:407] Connection header received
0923 20:27:26.000002 (+ 2985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.000007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.000080 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.000161 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.002359 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.003141 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.005621 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.006031 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.006131 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.022991 (+ 16860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.022999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.023004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.023059 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.027376 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.028464 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.028471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.028474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.028625 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.029219 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.029224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.029227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.029675 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.029842 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.030961 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.031337 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:27:26.033252 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:25.995981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:40631 (local address 127.0.0.1:56272)
0923 20:27:25.996837 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:25.996891 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:25.999679 (+ 2788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:26.000456 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:26.000472 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:26.001332 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:26.002158 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.002185 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.003312 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.003320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.005451 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.005471 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.006274 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.006284 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.006741 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.018943 (+ 12202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:26.018989 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:26.022604 (+ 3615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:26.027548 (+ 4944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.027558 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.027577 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.028312 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.028748 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.028755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.028760 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.029053 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.029825 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:26.029833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:26.030376 (+ 543us) client_negotiation.cc:769] Sending connection context
0923 20:27:26.030807 (+ 431us) client_negotiation.cc:240] Negotiation successful
0923 20:27:26.031392 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":310,"threads_started":1}
I20260923 20:27:26.035450 5326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195245, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.127.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab.
Entry for principal kudu/127.14.127.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab.
I20260923 20:27:26.052052 14847 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFITuhD/build/asan/bin/kudu
/tmp/dist-test-taskFITuhD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.252:39459
--webserver_interface=127.14.127.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab
--principal=kudu/127.14.127.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.127.212:42823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.127.254:40631,127.14.127.253:45227,127.14.127.252:39459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:27:26.053632 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.036668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.14.127.253:45227)
0923 20:27:26.037507 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.037515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.037555 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:27:26.037631 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.037640 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.037910 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.038745 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.039528 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.040646 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.042278 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.043317 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.043447 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.047436 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.047469 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.047491 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.047532 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.050528 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.051352 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.051359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.051362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.051484 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.051918 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.051921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.051923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.052367 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.052526 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.052984 (+ 458us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.053322 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":642,"thread_start_us":504,"threads_started":1}
I20260923 20:27:26.053709 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.036370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45756)
0923 20:27:26.036541 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.036564 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:26.036677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:26.038090 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:26.038098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:26.038512 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:26.038770 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.038786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.040785 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.040793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.042080 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.042095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.043508 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.043518 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.043794 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.044452 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:26.044482 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:26.047217 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:26.050706 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.050711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.050716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.051220 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.051604 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.051611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.051616 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.051803 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.052497 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:26.052505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:26.052707 (+ 202us) client_negotiation.cc:769] Sending connection context
0923 20:27:26.052868 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 20:27:26.053205 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:27:26.056983 5326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:27:26.058238 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.057642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42512)
0923 20:27:26.057844 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.058022 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 20:27:26.058503 5275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
I20260923 20:27:26.060088 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.059698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42518)
0923 20:27:26.059885 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.059979 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:27:26.060303 5326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
I20260923 20:27:26.109619 5326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 1
I20260923 20:27:26.110811 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.110287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42520)
0923 20:27:26.110490 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.110601 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:27:26.111881 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.111522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42534)
0923 20:27:26.111703 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.111784 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 20:27:26.112077 5326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
I20260923 20:27:26.170127 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } attempt: 5
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195245, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:27:26.180290 5326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 2
I20260923 20:27:26.181421 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.180877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42550)
0923 20:27:26.181105 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.181214 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 20:27:26.182482 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.182103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42554)
0923 20:27:26.182254 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.182338 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:27:26.182858 5326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
I20260923 20:27:26.185108 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.171060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.14.127.253:45227)
0923 20:27:26.171231 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.171237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.171261 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:27:26.171403 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.171409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.171506 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.171640 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.172536 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.173801 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.175015 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.175654 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.175794 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.179097 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.179102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.179105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.179139 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.182462 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.183290 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.183296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.183299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.183435 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.183819 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.183823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.183825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.184090 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.184239 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.184611 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.184825 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:27:26.185154 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.170956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45768)
0923 20:27:26.171155 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.171182 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:26.171293 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:26.171674 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:26.171678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:26.172129 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:26.172340 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.172351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.173948 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.173951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.174865 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.174877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.175815 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.175819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.176021 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.176656 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:26.176674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:26.178916 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:26.182597 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.182602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.182607 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.183167 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.183551 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.183556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.183560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.183701 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.184206 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:26.184211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:26.184364 (+ 153us) client_negotiation.cc:769] Sending connection context
0923 20:27:26.184501 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:27:26.184790 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 20:27:26.186118 5252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:27:26.187251 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.186815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42568)
0923 20:27:26.187004 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.187095 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 20:27:26.188254 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.187854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42580)
0923 20:27:26.188015 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.188113 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:27:26.188503 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
I20260923 20:27:26.239743 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 1
I20260923 20:27:26.240947 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.240417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42596)
0923 20:27:26.240636 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.240746 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:27:26.242043 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.241605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42612)
0923 20:27:26.241784 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.241888 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:27:26.242280 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
I20260923 20:27:26.274094 5326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 3
I20260923 20:27:26.275439 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.274828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42620)
0923 20:27:26.275098 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.275219 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:27:26.276669 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.276107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42632)
0923 20:27:26.276382 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.276511 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260923 20:27:26.276947 5326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
I20260923 20:27:26.310527 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 2
I20260923 20:27:26.311823 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.311222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42646)
0923 20:27:26.311486 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.311603 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:27:26.312947 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.312532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42652)
0923 20:27:26.312691 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.312796 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:27:26.313169 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
W20260923 20:27:26.383760 5333 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:27:26.384114 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:27:26.384181 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:27:26.384254 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:27:26.391613 5333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:27:26.391705 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:27:26.391767 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:27:26.391817 5333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:27:26.391880 5333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:27:26.401443 5333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.127.212:42823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.127.254:40631,127.14.127.253:45227,127.14.127.252:39459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/krb5kdc/kudu_127.14.127.252.keytab
--principal=kudu/127.14.127.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.127.252:39459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.127.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:27:26.402667 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:27:26.403924 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:27:26.415519 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:26.415619 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:27:26.415652 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:27:26.416692 5333 server_base.cc:1034] running on GCE node
I20260923 20:27:26.417469 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:27:26.419268 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:27:26.419392 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 3
I20260923 20:27:26.420154 5326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 4
I20260923 20:27:26.420562 5333 hybrid_clock.cc:648] HybridClock initialized: now 1790195246420467 us; error 47 us; skew 500 ppm
I20260923 20:27:26.420949 5333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:27:26.421599 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.420800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42666)
0923 20:27:26.421143 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.421314 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 20:27:26.422894 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.422423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42680)
0923 20:27:26.422680 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.422768 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 20:27:26.423004 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.420052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42662)
0923 20:27:26.422680 (+ 2628us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.422796 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2401}
W20260923 20:27:26.423158 5326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
I20260923 20:27:26.424175 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.423762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42686)
0923 20:27:26.423945 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.424044 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260923 20:27:26.424376 5252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.127.252:39459: Network error: Client connection negotiation failed: client connection to 127.14.127.252:39459: connect: Connection refused (error 111)
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195246, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:27:26.425446 5333 init.cc:377] Logged in from keytab as kudu/127.14.127.252@KRBTEST.COM (short username kudu)
I20260923 20:27:26.427138 5333 webserver.cc:460] Webserver started at http://127.14.127.252:35825/ using document root <none> and password file <none>
I20260923 20:27:26.428105 5333 fs_manager.cc:359] Metadata directory not provided
I20260923 20:27:26.428226 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:27:26.428514 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:27:26.431510 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/data/instance:
uuid: "34e5be2012444eb79dc129b56b00c6e6"
format_stamp: "Formatted at 2026-09-23 20:27:26 on dist-test-slave-wmd0"
server_key: "cef7b888ee8db3de13ec71e04e7aff22"
server_key_iv: "034ff06d4dcc68ff59bb0ef62daa1339"
server_key_version: "encryptionkey@0"
I20260923 20:27:26.432243 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/wal/instance:
uuid: "34e5be2012444eb79dc129b56b00c6e6"
format_stamp: "Formatted at 2026-09-23 20:27:26 on dist-test-slave-wmd0"
server_key: "cef7b888ee8db3de13ec71e04e7aff22"
server_key_iv: "034ff06d4dcc68ff59bb0ef62daa1339"
server_key_version: "encryptionkey@0"
I20260923 20:27:26.437294 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:27:26.440704 5348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:27:26.441959 5333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:27:26.442133 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/data,/tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/wal
uuid: "34e5be2012444eb79dc129b56b00c6e6"
format_stamp: "Formatted at 2026-09-23 20:27:26 on dist-test-slave-wmd0"
server_key: "cef7b888ee8db3de13ec71e04e7aff22"
server_key_iv: "034ff06d4dcc68ff59bb0ef62daa1339"
server_key_version: "encryptionkey@0"
I20260923 20:27:26.442341 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:27:26.459338 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:27:26.463208 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:27:26.463469 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:27:26.482956 5333 rpc_server.cc:307] RPC server started. Bound to: 127.14.127.252:39459
I20260923 20:27:26.482963 5399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.127.252:39459 every 8 connection(s)
I20260923 20:27:26.484263 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/data/info.pb
I20260923 20:27:26.487694 14847 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFITuhD/build/asan/bin/kudu as pid 5333
I20260923 20:27:26.487890 14847 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFITuhD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194882707199-14847-0/minicluster-data/master-2/wal/instance
I20260923 20:27:26.488128 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:27:26.488468 14847 external_mini_cluster.cc:1428] Setting key e4dd92a2c4a799f439c65bca6450d508
I20260923 20:27:26.496737 5400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195246, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
I20260923 20:27:26.510704 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.490367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.14.127.252:39459)
0923 20:27:26.491151 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.491162 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.491203 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:27:26.492817 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.492837 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.493214 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.493592 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.494992 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.496208 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.497450 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.498131 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.498315 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.502682 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.502693 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.502706 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.502751 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.506669 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.507468 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.507474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.507481 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.507632 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.508005 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.508011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.508014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.508525 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.508827 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.509261 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.509671 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":360,"threads_started":1}
I20260923 20:27:26.514266 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.499149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.14.127.254:40631)
0923 20:27:26.499361 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.499370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.500056 (+ 686us) server_negotiation.cc:407] Connection header received
0923 20:27:26.500236 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.500243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.500334 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.500460 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.501425 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.502157 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.504089 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.504693 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.504791 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.508497 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.508502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.508505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.508541 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.511416 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.512111 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.512116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.512119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.512266 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.512639 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.512645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.512648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.512987 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.513156 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.513703 (+ 547us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.513956 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:27:26.515288 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.499159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.254:40631 (local address 127.0.0.1:56280)
0923 20:27:26.499906 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.499952 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:26.500098 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:26.500495 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:26.500499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:26.501042 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:26.501279 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.501287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.502300 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.502305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.503937 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.503952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.504877 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.504886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.505083 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.505657 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:26.505679 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:26.508301 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:26.511555 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.511565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.511572 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.511983 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.512378 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.512383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.512387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.512528 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.513121 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:26.513131 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:26.513348 (+ 217us) client_negotiation.cc:769] Sending connection context
0923 20:27:26.514460 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0923 20:27:26.514839 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":386,"threads_started":1}
I20260923 20:27:26.516824 5400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.254@KRBTEST.COM
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195246, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:27:26.528744 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.513534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.14.127.254:40631)
0923 20:27:26.514146 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.514150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.514168 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:27:26.514205 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.514211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.514266 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.514387 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.515962 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.517008 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.518455 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.518926 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.519056 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.523101 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.523106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.523109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.523138 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.525986 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.526853 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.526857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.526859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.526970 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.527364 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.527367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.527369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.527617 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.527810 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.528180 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.528392 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":282,"threads_started":1}
I20260923 20:27:26.530570 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.517642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.253:45227 (local address 127.0.0.1:45780)
0923 20:27:26.517938 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.517953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:26.518078 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:26.518394 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:26.518399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:26.518849 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:26.518987 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.518994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.520087 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.520095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.520963 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.520974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.521657 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.521662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.521851 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.522374 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:26.522391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:26.524472 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:26.527801 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.527806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.527810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.528304 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.528781 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.528784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.528787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.528985 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.529626 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:26.529633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:26.529818 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 20:27:26.529937 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 20:27:26.530247 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:27:26.533208 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.517733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.14.127.253:45227)
0923 20:27:26.517879 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.517883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.518020 (+ 137us) server_negotiation.cc:407] Connection header received
0923 20:27:26.518201 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.518211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.518275 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.518372 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.519130 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.519945 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.521120 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.521505 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.521629 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.524665 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.524670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.524673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.524705 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.527632 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.528450 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.528457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.528460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.528626 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.529103 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.529111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.529116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.529455 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.529601 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.530060 (+ 459us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.532931 (+ 2871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 20:27:26.534461 5400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195246, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.252@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.127.253@KRBTEST.COM
I20260923 20:27:26.548365 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.535200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42696)
0923 20:27:26.535351 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.535369 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:26.535480 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:26.535786 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:26.535791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:26.536260 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:26.536422 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.536431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.537771 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.537774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.538995 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.539008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.539917 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.539921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.540140 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.540643 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:26.540659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:26.542575 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:26.546073 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.546078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.546082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.546450 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.546887 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.546893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.546897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.547023 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.547628 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:26.547632 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:26.547760 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 20:27:26.547869 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:27:26.548081 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":20}
I20260923 20:27:26.548451 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.535393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.14.127.252:39459)
0923 20:27:26.535562 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.535567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.535579 (+ 12us) server_negotiation.cc:407] Connection header received
0923 20:27:26.535615 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.535618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.535668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.535739 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.536552 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.537638 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.539136 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.539758 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.539896 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.542755 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.542760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.542763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.542789 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.545903 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.546584 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.546590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.546593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.546788 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.547133 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.547138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.547141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.547507 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.547648 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.547957 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.548209 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":76}
I20260923 20:27:26.550496 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.530347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.14.127.253:45227)
0923 20:27:26.533461 (+ 3114us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.533468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.533495 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:27:26.533544 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.533549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.533615 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.533719 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.535473 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.536546 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.538131 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.538795 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.538924 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.543495 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.543504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.543510 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.543558 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.546663 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.547316 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.547321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.547325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.547506 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.548972 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.548980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.548984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.549281 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.549488 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.549922 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.550135 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3010,"thread_start_us":3606,"threads_started":1}
I20260923 20:27:26.554035 5400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: Bootstrap starting.
I20260923 20:27:26.555076 14847 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:27:26.557372 5400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:26.558400 5400 log.cc:824] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:26.557269 5227 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9e9b743cc98a423788792c2ef7fd453b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56292
I20260923 20:27:26.561211 5400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: No bootstrap required, opened a new log
I20260923 20:27:26.568059 5400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } }
I20260923 20:27:26.568377 5400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e5be2012444eb79dc129b56b00c6e6, State: Initialized, Role: FOLLOWER
I20260923 20:27:26.568795 5400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } }
I20260923 20:27:26.569763 5408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.570173 5408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:26.570818 5400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:27:26.579856 5419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:27:26.580015 5419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:27:26.577812 5333 master_runner.cc:186] Error getting master registration for 127.14.127.254:40631: OK, instance_id { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" instance_seqno: 1790195245460394 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:27:26.589032 5333 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:27:26.601596 5252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 4
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195245, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.254@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
I20260923 20:27:26.616992 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.602325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42698)
0923 20:27:26.602593 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.602644 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:26.602776 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:26.603130 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:26.603135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:26.603544 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:26.603727 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.603735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.604761 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.604768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.605717 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.605728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.606464 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.606469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.607752 (+ 1283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.608368 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:26.608386 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:26.610739 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:26.614541 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.614546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.614557 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.614977 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.615339 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.615345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.615349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.615489 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.616019 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:26.616024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:26.616192 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 20:27:26.616329 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:27:26.616621 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260923 20:27:26.617007 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.602416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.14.127.252:39459)
0923 20:27:26.602643 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.602650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.602708 (+ 58us) server_negotiation.cc:407] Connection header received
0923 20:27:26.602916 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.602924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.602994 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.603104 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.603866 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.604617 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.605857 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.606317 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.606456 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.610917 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.610922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.610927 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.610962 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.614416 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.615093 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.615099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.615103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.615229 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.615603 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.615607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.615611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.615902 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.616078 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.616440 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.616636 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:27:26.623018 5252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b: Bootstrap starting.
I20260923 20:27:26.627280 5252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:26.628604 5252 log.cc:824] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:26.632263 5252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b: No bootstrap required, opened a new log
I20260923 20:27:26.639833 5252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } }
I20260923 20:27:26.640254 5252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9b743cc98a423788792c2ef7fd453b, State: Initialized, Role: FOLLOWER
I20260923 20:27:26.640758 5252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } }
I20260923 20:27:26.641726 5422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.642117 5422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:26.643038 5252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:26.643105 5227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 20:27:26.643328 5227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:26.646487 5227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } }
I20260923 20:27:26.648016 5227 leader_election.cc:288] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [CANDIDATE]: Term 1 election: Requested vote from peers dfee02eb173141f3907249cf9f13d36d (127.14.127.253:45227), 34e5be2012444eb79dc129b56b00c6e6 (127.14.127.252:39459)
I20260923 20:27:26.649258 5375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e9b743cc98a423788792c2ef7fd453b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "34e5be2012444eb79dc129b56b00c6e6"
I20260923 20:27:26.649827 5375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:26.654500 5375 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e9b743cc98a423788792c2ef7fd453b in term 1.
I20260923 20:27:26.656270 5201 leader_election.cc:302] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e5be2012444eb79dc129b56b00c6e6, 9e9b743cc98a423788792c2ef7fd453b; no voters:
I20260923 20:27:26.656920 5422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:27:26.662887 5422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [term 1 LEADER]: Becoming Leader. State: Replica: 9e9b743cc98a423788792c2ef7fd453b, State: Running, Role: LEADER
I20260923 20:27:26.663686 5422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } }
I20260923 20:27:26.679775 5434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e9b743cc98a423788792c2ef7fd453b. Latest consensus state: current_term: 1 leader_uuid: "9e9b743cc98a423788792c2ef7fd453b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.680231 5434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: This master's current role is: LEADER
W20260923 20:27:26.699687 5206 tablet.cc:2369] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:27:26.714481 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.672568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.14.127.252:39459)
0923 20:27:26.672827 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.672836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.690483 (+ 17647us) server_negotiation.cc:407] Connection header received
0923 20:27:26.690709 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.690715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.690840 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.690937 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.693107 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.694226 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.696204 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.696920 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.697043 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.702765 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.702777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.702782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.702829 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.707890 (+ 5061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.709246 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.709279 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.709284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.709553 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.710288 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.710293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.710297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.710917 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.711361 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.712422 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.713166 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260923 20:27:26.715478 5354 tablet.cc:2369] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:27:26.718922 5443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:27:26.719049 5443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:27:26.719280 5442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:27:26.722323 5326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } attempt: 5
I20260923 20:27:26.724080 5442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:27:26.726277 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.669334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.14.127.253:45227)
0923 20:27:26.669510 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.669516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.672832 (+ 3316us) server_negotiation.cc:407] Connection header received
0923 20:27:26.673628 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.673635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.673722 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.673796 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.684564 (+ 10768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.685989 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.691645 (+ 5656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.692572 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.692703 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.706090 (+ 13387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.706101 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.706108 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.706168 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.716385 (+ 10217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.723593 (+ 7208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.723600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.723604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.723822 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.724304 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.724309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.724313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.724701 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.724940 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.725628 (+ 688us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.725889 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:27:26.711051 5185 master_runner.cc:186] Error getting master registration for 127.14.127.253:45227: OK, instance_id { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" instance_seqno: 1790195245971359 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Sep 23 20:27:26 dist-test-slave-wmd0 krb5kdc[5172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790195245, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.127.253@KRBTEST.COM for kudu/127.14.127.252@KRBTEST.COM
I20260923 20:27:26.742653 5375 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [term 1 FOLLOWER]: Refusing update from remote peer 9e9b743cc98a423788792c2ef7fd453b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:27:26.733510 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.668816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.14.127.254:40631)
0923 20:27:26.669033 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.669040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.680103 (+ 11063us) server_negotiation.cc:407] Connection header received
0923 20:27:26.681038 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.681047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.681174 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.681265 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.694015 (+ 12750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.695419 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.697708 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.698501 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.698645 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.723823 (+ 25178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.723831 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.723835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.723879 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.728580 (+ 4701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.729319 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.729324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.729327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.729476 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.731232 (+ 1756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.731238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.731242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.731624 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.731858 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.732736 (+ 878us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.733074 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:27:26.743777 5434 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:27:26.745014 5185 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:27:26.752561 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.722942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.127.252:39459 (local address 127.0.0.1:42714)
0923 20:27:26.723285 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 20:27:26.723303 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:27:26.723409 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:27:26.728337 (+ 4928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:27:26.728343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:27:26.728827 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:27:26.729034 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.729045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.736584 (+ 7539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.736590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.738064 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:27:26.738081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.739201 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.739207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:27:26.739460 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.740132 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:27:26.740158 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:27:26.743579 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:27:26.749190 (+ 5611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.749196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.749200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.749721 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.750688 (+ 967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:27:26.750694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:27:26.750697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:27:26.750861 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:27:26.751516 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:27:26.751521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:27:26.751711 (+ 190us) client_negotiation.cc:769] Sending connection context
0923 20:27:26.751859 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 20:27:26.752176 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260923 20:27:26.753974 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:26.723638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.14.127.252:39459)
0923 20:27:26.726657 (+ 3019us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:26.726664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:26.726854 (+ 190us) server_negotiation.cc:407] Connection header received
0923 20:27:26.727252 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:26.727259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:26.727670 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:26.735125 (+ 7455us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:26.735295 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.736396 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.738277 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:26.739018 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:26.743823 (+ 4805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:26.744026 (+ 203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:27:26.744031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:27:26.744036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:27:26.744079 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:27:26.748928 (+ 4849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.750378 (+ 1450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.750383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.750387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.750525 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:27:26.751003 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:27:26.751008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:27:26.751013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:27:26.751372 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:27:26.753098 (+ 1726us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:26.753222 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 20:27:26.753559 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2385}
I20260923 20:27:26.760272 5326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d: Bootstrap starting.
I20260923 20:27:26.764442 5326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d: Neither blocks nor log segments found. Creating new log.
I20260923 20:27:26.765897 5326 log.cc:824] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:27:26.773165 5326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d: No bootstrap required, opened a new log
I20260923 20:27:26.785847 5408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e9b743cc98a423788792c2ef7fd453b. Latest consensus state: current_term: 1 leader_uuid: "9e9b743cc98a423788792c2ef7fd453b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.786170 5408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:26.786964 5445 mvcc.cc:205] Tried to move back new op lower bound from 7332639730644103168 to 7332639730341593088. Current Snapshot: MvccSnapshot[applied={T|T < 7332639730644103168}]
I20260923 20:27:26.793738 5326 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } }
I20260923 20:27:26.794252 5326 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfee02eb173141f3907249cf9f13d36d, State: Initialized, Role: FOLLOWER
I20260923 20:27:26.794883 5326 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } }
I20260923 20:27:26.802771 5422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e9b743cc98a423788792c2ef7fd453b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.803135 5422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:26.807174 5450 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.807641 5450 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:26.808291 5408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e9b743cc98a423788792c2ef7fd453b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.808588 5408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:26.810652 5442 catalog_manager.cc:1347] Generated new cluster ID: dcaab39a9fd9467cbf97a2e083e23c37
I20260923 20:27:26.810804 5442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:27:26.812352 5326 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:27:26.812944 5300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:27:26.812345 5301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e9b743cc98a423788792c2ef7fd453b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dfee02eb173141f3907249cf9f13d36d"
I20260923 20:27:26.819911 5300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [term 1 FOLLOWER]: Refusing update from remote peer 9e9b743cc98a423788792c2ef7fd453b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:27:26.824754 5301 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e9b743cc98a423788792c2ef7fd453b in term 1.
I20260923 20:27:26.829092 5434 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:27:26.866994 5442 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:27:26.868921 5442 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:27:26.871536 5456 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e9b743cc98a423788792c2ef7fd453b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.872031 5456 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:26.872495 5450 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e9b743cc98a423788792c2ef7fd453b. Latest consensus state: current_term: 1 leader_uuid: "9e9b743cc98a423788792c2ef7fd453b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.872792 5450 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:27:26.874001 5422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e9b743cc98a423788792c2ef7fd453b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9b743cc98a423788792c2ef7fd453b" member_type: VOTER last_known_addr { host: "127.14.127.254" port: 40631 } } peers { permanent_uuid: "dfee02eb173141f3907249cf9f13d36d" member_type: VOTER last_known_addr { host: "127.14.127.253" port: 45227 } } peers { permanent_uuid: "34e5be2012444eb79dc129b56b00c6e6" member_type: VOTER last_known_addr { host: "127.14.127.252" port: 39459 } } }
I20260923 20:27:26.874308 5422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b [sys.catalog]: This master's current role is: LEADER
I20260923 20:27:26.890769 5442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e9b743cc98a423788792c2ef7fd453b: Generated new TSK 0
I20260923 20:27:26.892043 5442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:27:26.920451 5468 catalog_manager.cc:1259] Loaded cluster ID: dcaab39a9fd9467cbf97a2e083e23c37
I20260923 20:27:26.920578 5468 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d: loading cluster ID for follower catalog manager: success
I20260923 20:27:26.925913 5468 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d: acquiring CA information for follower catalog manager: success
I20260923 20:27:26.932101 5468 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dfee02eb173141f3907249cf9f13d36d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 20:27:27.047264 5331 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.031889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.14.127.253:45227)
0923 20:27:27.032136 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.032143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.041166 (+ 9023us) server_negotiation.cc:407] Connection header received
0923 20:27:27.044048 (+ 2882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.044055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.044153 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.044231 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.047011 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45790: BlockingRecv error: recv got EOF from 127.0.0.1:45790 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:27:27.048930 5405 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.038992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.14.127.254:40631)
0923 20:27:27.039202 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.039208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.045240 (+ 6032us) server_negotiation.cc:407] Connection header received
0923 20:27:27.045933 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.045939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.046030 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.046103 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.048679 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56304: BlockingRecv error: recv got EOF from 127.0.0.1:56304 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 20:27:27.050815 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.041777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.14.127.252:39459)
0923 20:27:27.041974 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.041981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.045478 (+ 3497us) server_negotiation.cc:407] Connection header received
0923 20:27:27.049369 (+ 3891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.049375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.049458 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.049533 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.050562 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42720: BlockingRecv error: recv got EOF from 127.0.0.1:42720 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:27:27.103861 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.087175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.14.127.253:45227)
0923 20:27:27.087363 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.087370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.096075 (+ 8705us) server_negotiation.cc:407] Connection header received
0923 20:27:27.096417 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.096424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.096635 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.096777 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.097489 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.098765 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.100822 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.101515 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.101608 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.102341 (+ 733us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.103053 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.103164 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.103399 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.103500 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 20:27:27.148799 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.136033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.14.127.252:39459)
0923 20:27:27.136198 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.136205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.146696 (+ 10491us) server_negotiation.cc:407] Connection header received
0923 20:27:27.147155 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.147161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.147324 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.147412 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.148545 (+ 1133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42736: BlockingRecv error: recv got EOF from 127.0.0.1:42736 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:27:27.197690 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.183468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.14.127.253:45227)
0923 20:27:27.183665 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.183671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.191263 (+ 7592us) server_negotiation.cc:407] Connection header received
0923 20:27:27.191483 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.191491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.191602 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.191716 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.192220 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.193414 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.195455 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.196200 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.196335 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.196828 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.197026 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.197120 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.197284 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.197366 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 20:27:27.219662 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.209085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.14.127.252:39459)
0923 20:27:27.209274 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.209280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.217771 (+ 8491us) server_negotiation.cc:407] Connection header received
0923 20:27:27.218084 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.218091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.218189 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.218274 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.219418 (+ 1144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42748: BlockingRecv error: recv got EOF from 127.0.0.1:42748 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:27:27.232581 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.225360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.14.127.253:45227)
0923 20:27:27.225555 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.225562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.226113 (+ 551us) server_negotiation.cc:407] Connection header received
0923 20:27:27.226318 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.226324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.226424 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.226522 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.227091 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.228343 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.230476 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.231171 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.231303 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.231757 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.231916 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.232000 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.232214 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.232288 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 20:27:27.241326 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.239001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.14.127.252:39459)
0923 20:27:27.239180 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.239186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.239757 (+ 571us) server_negotiation.cc:407] Connection header received
0923 20:27:27.239985 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.239992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.240090 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.240189 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.241092 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42750: BlockingRecv error: recv got EOF from 127.0.0.1:42750 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:27:27.256016 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.248902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.14.127.253:45227)
0923 20:27:27.249125 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.249131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.249657 (+ 526us) server_negotiation.cc:407] Connection header received
0923 20:27:27.249840 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.249847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.249962 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.250082 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.250564 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.251754 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.253722 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.254499 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.254669 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.255202 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.255344 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.255445 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.255613 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.255703 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260923 20:27:27.265211 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.262874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.14.127.252:39459)
0923 20:27:27.263158 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.263164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.263688 (+ 524us) server_negotiation.cc:407] Connection header received
0923 20:27:27.263888 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.263893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.264005 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.264126 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.264976 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42758: BlockingRecv error: recv got EOF from 127.0.0.1:42758 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
I20260923 20:27:27.283970 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.276874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.14.127.253:45227)
0923 20:27:27.277092 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.277100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.277621 (+ 521us) server_negotiation.cc:407] Connection header received
0923 20:27:27.277843 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.277853 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.277946 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.278054 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.278570 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.279719 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.281711 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.282380 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.282514 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.283103 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.283249 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.283347 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.283536 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.283616 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 20:27:27.299153 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.296749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.14.127.252:39459)
0923 20:27:27.296955 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.296962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.297556 (+ 594us) server_negotiation.cc:407] Connection header received
0923 20:27:27.297778 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.297785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.297893 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.298013 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.298900 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42766: BlockingRecv error: recv got EOF from 127.0.0.1:42766 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:27:27.326300 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.319143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.14.127.253:45227)
0923 20:27:27.319334 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.319341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.319904 (+ 563us) server_negotiation.cc:407] Connection header received
0923 20:27:27.320114 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.320122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.320219 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.320327 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.320829 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.322175 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.324169 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.324867 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.325003 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.325535 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.325678 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.325762 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.325938 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.326032 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 20:27:27.347092 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.344583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.14.127.252:39459)
0923 20:27:27.344779 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.344785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.345345 (+ 560us) server_negotiation.cc:407] Connection header received
0923 20:27:27.345551 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.345557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.345655 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.345766 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.346866 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42774: BlockingRecv error: recv got EOF from 127.0.0.1:42774 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:27:27.391419 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.384947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.14.127.253:45227)
0923 20:27:27.385196 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.385203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.385584 (+ 381us) server_negotiation.cc:407] Connection header received
0923 20:27:27.385867 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.385874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.385983 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.386055 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.386732 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.387572 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.389503 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.389956 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.390061 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.390550 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.390719 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.390792 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.390983 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.391047 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 20:27:27.402261 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.400457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.14.127.252:39459)
0923 20:27:27.400649 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.400655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.400795 (+ 140us) server_negotiation.cc:407] Connection header received
0923 20:27:27.401020 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.401026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.401117 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.401219 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.402007 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42788: BlockingRecv error: recv got EOF from 127.0.0.1:42788 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:27:27.475875 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.470403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.14.127.253:45227)
0923 20:27:27.470664 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.470673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.471025 (+ 352us) server_negotiation.cc:407] Connection header received
0923 20:27:27.471295 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.471299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.471424 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.471518 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.471936 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.472744 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.474209 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.474602 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.474783 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.475188 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.475289 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.475382 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.475535 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.475611 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 20:27:27.483336 5403 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:27:27.481265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.14.127.252:39459)
0923 20:27:27.481506 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.481512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.481846 (+ 334us) server_negotiation.cc:407] Connection header received
0923 20:27:27.482006 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.482009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.482125 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.482242 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:27:27.483123 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42802: BlockingRecv error: recv got EOF from 127.0.0.1:42802 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:27:27.582028 5419 catalog_manager.cc:1259] Loaded cluster ID: dcaab39a9fd9467cbf97a2e083e23c37
I20260923 20:27:27.582180 5419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: loading cluster ID for follower catalog manager: success
I20260923 20:27:27.584702 5419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: acquiring CA information for follower catalog manager: success
I20260923 20:27:27.586481 5419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34e5be2012444eb79dc129b56b00c6e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:27:27.620517 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.615472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.14.127.253:45227)
0923 20:27:27.615699 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.615706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.616006 (+ 300us) server_negotiation.cc:407] Connection header received
0923 20:27:27.616247 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.616252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.616323 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.616420 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.616829 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.617507 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.618860 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.619269 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.619406 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.619806 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.619915 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.620012 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.620160 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.620239 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:27:27.632201 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.626707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.14.127.252:39459)
0923 20:27:27.626927 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.626933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.627174 (+ 241us) server_negotiation.cc:407] Connection header received
0923 20:27:27.627303 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.627307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.627453 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.627560 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.627915 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.628735 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.630122 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.630540 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.630700 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.631108 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.631619 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.631738 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.631798 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.631860 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 20:27:27.646664 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.639672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.14.127.252:39459)
0923 20:27:27.640043 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.640050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.640068 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:27:27.640146 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.640156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.640215 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.640327 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.641241 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.642210 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.644297 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.644973 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.645161 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.645870 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.646023 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.646128 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.646254 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.646331 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260923 20:27:27.646754 5331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.637967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.14.127.253:45227)
0923 20:27:27.638191 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.638200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.639406 (+ 1206us) server_negotiation.cc:407] Connection header received
0923 20:27:27.639586 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.639594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.639702 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.639805 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.641304 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.642062 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.644510 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.644957 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.645030 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.645919 (+ 889us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.646041 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.646144 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.646394 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.646475 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 20:27:27.647589 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:27:27.637800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.14.127.254:40631)
0923 20:27:27.639864 (+ 2064us) server_negotiation.cc:206] Beginning negotiation
0923 20:27:27.639876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:27:27.639892 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:27:27.639984 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:27:27.639987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:27:27.640127 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:27:27.640265 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:27:27.640855 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.641923 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.644382 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:27:27.645050 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:27:27.645202 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:27:27.646363 (+ 1161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:27:27.646928 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:27:27.647044 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:27:27.647192 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 20:27:27.647276 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1944,"thread_start_us":2597,"threads_started":1}
I20260923 20:27:27.652207 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 5185
I20260923 20:27:27.653155 5247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:27.763996 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 5185
I20260923 20:27:27.785066 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 5259
I20260923 20:27:27.786085 5321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:27.894532 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 5259
I20260923 20:27:27.914688 14847 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFITuhD/build/asan/bin/kudu pid 5333
I20260923 20:27:27.915766 5395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:27:28.024300 14847 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFITuhD/build/asan/bin/kudu with pid 5333
2026-09-23T20:27:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4343 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8731 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (365236 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST